Standalone backup agent for Microsoft Windows servers and workstations (formerly Veeam Endpoint Backup FREE)
Post Reply
mcz
Veeam Legend
Posts: 934
Liked: 214 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Location: Rheintal, Austria
Contact:

backup destroys network connection

Post by mcz »

Hi!

We've got a 1 Gbit Network and use our VAW to back up several Workstations and Notebooks. From time to time, certain workstations just have no network connection while the backup job is running - and I really mean that there's no network connection anymore, because even a ping won't be successful. The job itself would run fine and if you wait until it has finished or cancel it, network connection will be back. I've just seen this behaviour today and checked how much Data Veeam would process - UI told me that it would process 34 MB/s which is about 25 % of a Gigabit Network (our Servers where the veeam instance is running are connected with 10 Gbit/s) so I don't see a reason why this could happen.

2 hours later I took my notebook and connected it via cable when I realized that the ping latency in the local network got really ugly. First of all I suspected a potential network issue but then I saw that the veeam agent was running - same situation, allthough I had a higher throughput (but again, not close to 1 Gbit/s). When the job got finished, the ping went down to 1 ms or smaller (as expected).

Image

Is this a known issue, what is going on here?

Thanks in advance!
Dima P.
Product Manager
Posts: 14643
Liked: 1675 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: backup destroys network connection

Post by Dima P. »

Hello Michael,

1. Can you please clarify how throttling options (both network and backup activity) are set?
2. What backup repository you are using?

Cheers!
mcz
Veeam Legend
Posts: 934
Liked: 214 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Location: Rheintal, Austria
Contact:

Re: backup destroys network connection

Post by mcz »

Hi Dmitry,

thanks for your quick response. We do not have network throttling set and the repository is just a attached vmdk to our proxy server (ReFS).
Dima P.
Product Manager
Posts: 14643
Liked: 1675 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: backup destroys network connection

Post by Dima P. »

the repository is just a attached vmdk to our proxy server (ReFS)
How that is configured in Veeam B&R? As a Windows repository?
mcz
Veeam Legend
Posts: 934
Liked: 214 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Location: Rheintal, Austria
Contact:

Re: backup destroys network connection

Post by mcz »

Hi Dmitry,

correct, it's a windows repository.
Dima P.
Product Manager
Posts: 14643
Liked: 1675 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: backup destroys network connection

Post by Dima P. »

Michael,

Thanks! As a workaround you can throttle network traffic between agent and repository via Traffic Throttling in the Veeam B&R console.

As for network issues, despite the workaround, please raise a support case and let our team check the debug logs and share the case ID with us. Cheers!
mcz
Veeam Legend
Posts: 934
Liked: 214 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Location: Rheintal, Austria
Contact:

Re: backup destroys network connection

Post by mcz »

Hi Dmitry,

let's see what support finds out, I've created the case 03983354.

Thanks!
mcz
Veeam Legend
Posts: 934
Liked: 214 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Location: Rheintal, Austria
Contact:

Re: backup destroys network connection

Post by mcz »

Hi guys,

meanwhile I've created another case: #03983354. Actually it's quite simple: Support told me that the network is just the bottleneck, so there's only one way to limit the impact: throttle the traffic. But when I think about it, I don't like it due to the fact, that you cannot really predict how much you have to throttle. In our case we protect some notebooks and if they are connected via cable (1 Gbit) it would be eays to limit to say 800 Mbit/s. But what happens when the user picks his device and connects it via WiFi for an online-meeting? Depending on signal strength and so on, 800 Mbit/s would be too much.

Actually I'm surprised that it's even possible that a single process consumes the whole bandwidth/send buffers and that windows does not some kind of sharing of buffers across all processes. I mean it's clear that you wouldn't be able to do a good upload while the backup consumes almost the whole bandwidth, but the thing is that you really can't work at all on your device during the backup. For instance, last week I took my notebook and connected it via WiFi and the backup was running. During that period, I wasn't even able to connect to our firewall (HTML-Interface) and work properly on that interface when we all know that we really send very very little data to that endpoint. In some situations, even pings could result in timeouts and I assume that simply the ICMP output is discared due to the full send buffers.

Once again, it's not that you're limited a bit or that you "feel the upload" but you can't really use the network at all! I think the best way to fix this is to let the agent/OS decide, how much throughput it can use for being able to work properly at the same time. Trying to set some static limitations can (and will) help but depending on the situation it might be too much / too less. Feature request.

How does veeam think about it? Thanks!
Dima P.
Product Manager
Posts: 14643
Liked: 1675 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: backup destroys network connection

Post by Dima P. »

Hello Michael,

Thanks for sharing the update and sorry for the delayed response. We've discussed the possibility of similar functionality with RnD in the past and the trick is - to measure the latency and throttle it automatically you need to have a dedicated server which is not loaded with regular user activity and backup activity (otherwise, automated throttling would simply be ruined by incorrect bandwidth metrics and as a result it wont work). Honestly, I think the best answer here is tune the schedule - to make sure that users do not run backup jobs when they are actually working (i.e. events like lock or log off to triggers the backup). Another handy tool would be to disable backup over VPN/metered connections (you can possibility limit the WiFi networks as well).

For upcoming major version there would be a registry 'hack' to specify the exact adapters to use during the backup (and ignore others), that potentially can disable the backup over WiFi adapters.

Hope that helps!
ColonialsMZ
Lurker
Posts: 2
Liked: 1 time
Joined: Jan 08, 2021 2:36 pm
Full Name: Mike
Contact:

Re: backup destroys network connection

Post by ColonialsMZ »

Hello,

We're having a similar issue in our district. We have a combination of hardwired gigabit, wifi 400 megabit and wifi gigabit connections. We've enabled traffic throttling for the veeam project groups at different times to be 60-120 mb and still have each type of device lose internet connectivity during a backup. We can lower it further than 60 mb, but that's just far too slow of a backup process. We are backing up to NAS repositories on-site through our veeam proxy server. The NAS' are set up in Veeam B&R as SMB backup repositories, the proxy is set up as a Windows Repository. The servers and NAS' all use gigabit connections and the issue still occurs even if that's the only running veeam job.

Veeam support suggested looking at QOS settings on the firewall which we haven't gotten to yet, but reading the above thread makes it seem like this may be expected behavior? Before we start trying things out I'd like to try and find out more about whether our configuration is correct or if this is more or less the behavior we should expect. Is there anything we can try within veeam to keep that internet connection alive while the backup is running?

Thank you
mcz
Veeam Legend
Posts: 934
Liked: 214 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Location: Rheintal, Austria
Contact:

Re: backup destroys network connection

Post by mcz »

...that was also my thought above... I'm surprised that it's possible that a single process DoSes your local network connection... Is this really expected or maybe an design issue??
ColonialsMZ
Lurker
Posts: 2
Liked: 1 time
Joined: Jan 08, 2021 2:36 pm
Full Name: Mike
Contact:

Re: backup destroys network connection

Post by ColonialsMZ » 1 person likes this post

Update (potential solution): In our particular case, RAM usage seems to be the culprit. The Veeam agent was having issues connecting to the SQL LocalDB on the client machine. Logs and monitoring show an average of ~200MB of available RAM. By closing some unnecessary applications during the backup window and increasing the available RAM, we were able to have a week (and counting) of successful backups on the affected machine.
Egor Yakovlev
Product Manager
Posts: 2568
Liked: 700 times
Joined: Jun 14, 2013 9:30 am
Full Name: Egor Yakovlev
Location: Prague, Czech Republic
Contact:

Re: backup destroys network connection

Post by Egor Yakovlev »

Thanks for update and sharing solution!
Post Reply

Who is online

Users browsing this forum: No registered users and 10 guests