Comprehensive data protection for all workloads
Post Reply
Shogan

How does Veeam B&R handle network disconnections

Post by Shogan »

Hi guys,

A question about Veeam Backup - how does it handle network disconnections? i.e. between the source and target when doing a backup? My testing thus far is that even if a short disconnection occurs on the network, the backup fails. Is there any built in protection against this, or a certain number of retries that occur before the job fails?

Thanks!

foggy
Veeam Software
Posts: 19433
Liked: 1763 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: How does Veeam B&R handle network disconnections

Post by foggy »

Minor packet loss is not a problem, backup will still be able to proceed correctly, but in case of a tangible disconnection the job fails.

Shogan

Re: How does Veeam B&R handle network disconnections

Post by Shogan »

Thanks Foggy, so for example, a disconnection of 5 seconds would probably result in failure then? Most definitely if there was a disconnection of 30 seconds?

Reason I ask is we have a fibre connection that sometimes gets disconnections and needs to be failed over via an automated script - this causes a disconnection of around 30 seconds, meaning veeam jobs fail.

I (and I am sure others) would really like to see a feature in Veeam Backup that can handle disconnections like this.

Gostev
SVP, Product Management
Posts: 26679
Liked: 4268 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: How does Veeam B&R handle network disconnections

Post by Gostev »

Anything that causes TCP/IP connection to drop will result in job failure.

Shogan

Re: How does Veeam B&R handle network disconnections

Post by Shogan »

Hey Gostev,

Any plans for a new feature in the next release / update release of v6 to include an automatic dropped connection retry feature for jobs? :wink:

I would like to formerly request this feature!

Gostev
SVP, Product Management
Posts: 26679
Liked: 4268 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: How does Veeam B&R handle network disconnections

Post by Gostev »

Actually, we have job retry feature since v3. Job will automatically retry, and process only VMs which failed during main job run. This behavior is controlled on the same page where you set the job schedule. Thanks!

Shogan

Re: How does Veeam B&R handle network disconnections

Post by Shogan »

Thanks Anton,

Maybe I should have been more specific - I would like to request a feature where the current VM being backed up in a job will resume backing up if there is a drop in network connection, without the need for the VM to be retried again from the start. So if 20GB out of 50GB have been processed so far, and there is a drop, then the job tries to resume from that 20GB point and continue on with things.

Our issue is we back up between Data Centres using a fibre connection across the city, and sometimes there can be short periods of maintenance / times when the supplier re-route the connection and there is a 5 - 30 second drop in network connectivity. It would be a bit of a pain for large (500GB) VM backups to need to retry completely, rather than just resume. :)

Gostev
SVP, Product Management
Posts: 26679
Liked: 4268 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: How does Veeam B&R handle network disconnections

Post by Gostev »

Resume, may be... don't recall many requests for this feature though. Possibly because nowadays, most connections have a very high uptime, especially those enterprise class ones which cost a lot.

arsprod
Enthusiast
Posts: 86
Liked: never
Joined: Jan 21, 2011 6:09 pm
Full Name: Aaron Spiegel
Contact:

Re: How does Veeam B&R handle network disconnections

Post by arsprod »

True that most enterprise class san's have high uptime, but I (and I would guess other folks) are using other devices (like simple nas or software iscsi like starwind) that don't necessarily have high uptime. I'm having a heck of a time with Veeam jobs becoming corrupted because of disconnects. Re-running the job doesn't help because the interruption corrupts the whole job.

Vitaliy S.
Product Manager
Posts: 24243
Liked: 1859 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: How does Veeam B&R handle network disconnections

Post by Vitaliy S. »

Hello Aaron,

Out of curiosity how often do you have disconnects? Have you been able to track down the initial reason for that? Thanks

arsprod
Enthusiast
Posts: 86
Liked: never
Joined: Jan 21, 2011 6:09 pm
Full Name: Aaron Spiegel
Contact:

Re: How does Veeam B&R handle network disconnections

Post by arsprod »

well, it's a long story! At the moment at least a couple times a week. I've been working with support and through that process (over a couple months) we've determined it's the iscsi datastores disconnecting, which in turn corrupts the replica job and I have to delete all the history and start over. I was using a Windows 2008r2 server and MS iscsi. I recently removed all the MS stuff and installed Starwind which seems much more stable, but still getting disconnected. Just today I moved that backup SAN to the dedicated iscsi switch - we'll see if that helps. Sorry you asked?

Vitaliy S.
Product Manager
Posts: 24243
Liked: 1859 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: How does Veeam B&R handle network disconnections

Post by Vitaliy S. »

Ouch...that sounds painful. I'm not a storage guru, but trying a dedicated switch might help here.

st0623
Technology Partner
Posts: 27
Liked: never
Joined: Jun 08, 2010 9:01 pm
Full Name: Steve Thompson
Contact:

Re: How does Veeam B&R handle network disconnections

Post by st0623 »

A couple of HyperIP customers have asked us to tune our timeouts for network disconnects in the WAN. To date, if Veeam B&R is running over HyperIP, we can keep the session/job up for up to 3 seconds, without causing the backup job to fail. We haven't tried a longer disconnect, but could test it with you. 3 seconds is forever in the network world, but we are aware of some links disconnecting for 5 seconds or longer. Sometimes the nature of MPLS or cheaper public internet connections. I'd be happy to talk with you, Shogan, if you want to reach out via my contact info below.
Regards,
Steve Thompson
HyperIP team at NetEx Software
steve.thompson@netex.com
704.467.6749

Shogan

Re: How does Veeam B&R handle network disconnections

Post by Shogan »

st0623 wrote:A couple of HyperIP customers have asked us to tune our timeouts for network disconnects in the WAN. To date, if Veeam B&R is running over HyperIP, we can keep the session/job up for up to 3 seconds, without causing the backup job to fail. We haven't tried a longer disconnect, but could test it with you. 3 seconds is forever in the network world, but we are aware of some links disconnecting for 5 seconds or longer. Sometimes the nature of MPLS or cheaper public internet connections. I'd be happy to talk with you, Shogan, if you want to reach out via my contact info below.
Hi Steve,

Thanks for your reply. At the moment we are running a script which monitors an end point on the other side - when this fails to ping a certain amount of times, it runs another script which fails over to a completely different supplier connection. This works out so that the disconnection is actually around 1 minute in some cases depending on the issue as the script needs time to ensure there is more than just a little bit of packet loss causing dropped pings before it switches over to the other connection.

I've noted down your details though in case things change here - I don't think much could be done about a long disconnection like the above :) In terms of Veeam though, we could really do with a feature that allows the job to resume / retry at times like these.

Post Reply

Who is online

Users browsing this forum: Bing [Bot], Google [Bot] and 29 guests