-
- Veeam Legend
- Posts: 934
- Liked: 214 times
- Joined: Jul 19, 2016 8:39 am
- Full Name: Michael
- Location: Rheintal, Austria
- Contact:
backup destroys network connection
Hi!
We've got a 1 Gbit Network and use our VAW to back up several Workstations and Notebooks. From time to time, certain workstations just have no network connection while the backup job is running - and I really mean that there's no network connection anymore, because even a ping won't be successful. The job itself would run fine and if you wait until it has finished or cancel it, network connection will be back. I've just seen this behaviour today and checked how much Data Veeam would process - UI told me that it would process 34 MB/s which is about 25 % of a Gigabit Network (our Servers where the veeam instance is running are connected with 10 Gbit/s) so I don't see a reason why this could happen.
2 hours later I took my notebook and connected it via cable when I realized that the ping latency in the local network got really ugly. First of all I suspected a potential network issue but then I saw that the veeam agent was running - same situation, allthough I had a higher throughput (but again, not close to 1 Gbit/s). When the job got finished, the ping went down to 1 ms or smaller (as expected).
Is this a known issue, what is going on here?
Thanks in advance!
We've got a 1 Gbit Network and use our VAW to back up several Workstations and Notebooks. From time to time, certain workstations just have no network connection while the backup job is running - and I really mean that there's no network connection anymore, because even a ping won't be successful. The job itself would run fine and if you wait until it has finished or cancel it, network connection will be back. I've just seen this behaviour today and checked how much Data Veeam would process - UI told me that it would process 34 MB/s which is about 25 % of a Gigabit Network (our Servers where the veeam instance is running are connected with 10 Gbit/s) so I don't see a reason why this could happen.
2 hours later I took my notebook and connected it via cable when I realized that the ping latency in the local network got really ugly. First of all I suspected a potential network issue but then I saw that the veeam agent was running - same situation, allthough I had a higher throughput (but again, not close to 1 Gbit/s). When the job got finished, the ping went down to 1 ms or smaller (as expected).
Is this a known issue, what is going on here?
Thanks in advance!
-
- Product Manager
- Posts: 14643
- Liked: 1675 times
- Joined: Feb 04, 2013 2:07 pm
- Full Name: Dmitry Popov
- Location: Prague
- Contact:
Re: backup destroys network connection
Hello Michael,
1. Can you please clarify how throttling options (both network and backup activity) are set?
2. What backup repository you are using?
Cheers!
1. Can you please clarify how throttling options (both network and backup activity) are set?
2. What backup repository you are using?
Cheers!
-
- Veeam Legend
- Posts: 934
- Liked: 214 times
- Joined: Jul 19, 2016 8:39 am
- Full Name: Michael
- Location: Rheintal, Austria
- Contact:
Re: backup destroys network connection
Hi Dmitry,
thanks for your quick response. We do not have network throttling set and the repository is just a attached vmdk to our proxy server (ReFS).
thanks for your quick response. We do not have network throttling set and the repository is just a attached vmdk to our proxy server (ReFS).
-
- Product Manager
- Posts: 14643
- Liked: 1675 times
- Joined: Feb 04, 2013 2:07 pm
- Full Name: Dmitry Popov
- Location: Prague
- Contact:
Re: backup destroys network connection
How that is configured in Veeam B&R? As a Windows repository?the repository is just a attached vmdk to our proxy server (ReFS)
-
- Veeam Legend
- Posts: 934
- Liked: 214 times
- Joined: Jul 19, 2016 8:39 am
- Full Name: Michael
- Location: Rheintal, Austria
- Contact:
Re: backup destroys network connection
Hi Dmitry,
correct, it's a windows repository.
correct, it's a windows repository.
-
- Product Manager
- Posts: 14643
- Liked: 1675 times
- Joined: Feb 04, 2013 2:07 pm
- Full Name: Dmitry Popov
- Location: Prague
- Contact:
Re: backup destroys network connection
Michael,
Thanks! As a workaround you can throttle network traffic between agent and repository via Traffic Throttling in the Veeam B&R console.
As for network issues, despite the workaround, please raise a support case and let our team check the debug logs and share the case ID with us. Cheers!
Thanks! As a workaround you can throttle network traffic between agent and repository via Traffic Throttling in the Veeam B&R console.
As for network issues, despite the workaround, please raise a support case and let our team check the debug logs and share the case ID with us. Cheers!
-
- Veeam Legend
- Posts: 934
- Liked: 214 times
- Joined: Jul 19, 2016 8:39 am
- Full Name: Michael
- Location: Rheintal, Austria
- Contact:
Re: backup destroys network connection
Hi Dmitry,
let's see what support finds out, I've created the case 03983354.
Thanks!
let's see what support finds out, I've created the case 03983354.
Thanks!
-
- Veeam Legend
- Posts: 934
- Liked: 214 times
- Joined: Jul 19, 2016 8:39 am
- Full Name: Michael
- Location: Rheintal, Austria
- Contact:
Re: backup destroys network connection
Hi guys,
meanwhile I've created another case: #03983354. Actually it's quite simple: Support told me that the network is just the bottleneck, so there's only one way to limit the impact: throttle the traffic. But when I think about it, I don't like it due to the fact, that you cannot really predict how much you have to throttle. In our case we protect some notebooks and if they are connected via cable (1 Gbit) it would be eays to limit to say 800 Mbit/s. But what happens when the user picks his device and connects it via WiFi for an online-meeting? Depending on signal strength and so on, 800 Mbit/s would be too much.
Actually I'm surprised that it's even possible that a single process consumes the whole bandwidth/send buffers and that windows does not some kind of sharing of buffers across all processes. I mean it's clear that you wouldn't be able to do a good upload while the backup consumes almost the whole bandwidth, but the thing is that you really can't work at all on your device during the backup. For instance, last week I took my notebook and connected it via WiFi and the backup was running. During that period, I wasn't even able to connect to our firewall (HTML-Interface) and work properly on that interface when we all know that we really send very very little data to that endpoint. In some situations, even pings could result in timeouts and I assume that simply the ICMP output is discared due to the full send buffers.
Once again, it's not that you're limited a bit or that you "feel the upload" but you can't really use the network at all! I think the best way to fix this is to let the agent/OS decide, how much throughput it can use for being able to work properly at the same time. Trying to set some static limitations can (and will) help but depending on the situation it might be too much / too less. Feature request.
How does veeam think about it? Thanks!
meanwhile I've created another case: #03983354. Actually it's quite simple: Support told me that the network is just the bottleneck, so there's only one way to limit the impact: throttle the traffic. But when I think about it, I don't like it due to the fact, that you cannot really predict how much you have to throttle. In our case we protect some notebooks and if they are connected via cable (1 Gbit) it would be eays to limit to say 800 Mbit/s. But what happens when the user picks his device and connects it via WiFi for an online-meeting? Depending on signal strength and so on, 800 Mbit/s would be too much.
Actually I'm surprised that it's even possible that a single process consumes the whole bandwidth/send buffers and that windows does not some kind of sharing of buffers across all processes. I mean it's clear that you wouldn't be able to do a good upload while the backup consumes almost the whole bandwidth, but the thing is that you really can't work at all on your device during the backup. For instance, last week I took my notebook and connected it via WiFi and the backup was running. During that period, I wasn't even able to connect to our firewall (HTML-Interface) and work properly on that interface when we all know that we really send very very little data to that endpoint. In some situations, even pings could result in timeouts and I assume that simply the ICMP output is discared due to the full send buffers.
Once again, it's not that you're limited a bit or that you "feel the upload" but you can't really use the network at all! I think the best way to fix this is to let the agent/OS decide, how much throughput it can use for being able to work properly at the same time. Trying to set some static limitations can (and will) help but depending on the situation it might be too much / too less. Feature request.
How does veeam think about it? Thanks!
-
- Product Manager
- Posts: 14643
- Liked: 1675 times
- Joined: Feb 04, 2013 2:07 pm
- Full Name: Dmitry Popov
- Location: Prague
- Contact:
Re: backup destroys network connection
Hello Michael,
Thanks for sharing the update and sorry for the delayed response. We've discussed the possibility of similar functionality with RnD in the past and the trick is - to measure the latency and throttle it automatically you need to have a dedicated server which is not loaded with regular user activity and backup activity (otherwise, automated throttling would simply be ruined by incorrect bandwidth metrics and as a result it wont work). Honestly, I think the best answer here is tune the schedule - to make sure that users do not run backup jobs when they are actually working (i.e. events like lock or log off to triggers the backup). Another handy tool would be to disable backup over VPN/metered connections (you can possibility limit the WiFi networks as well).
For upcoming major version there would be a registry 'hack' to specify the exact adapters to use during the backup (and ignore others), that potentially can disable the backup over WiFi adapters.
Hope that helps!
Thanks for sharing the update and sorry for the delayed response. We've discussed the possibility of similar functionality with RnD in the past and the trick is - to measure the latency and throttle it automatically you need to have a dedicated server which is not loaded with regular user activity and backup activity (otherwise, automated throttling would simply be ruined by incorrect bandwidth metrics and as a result it wont work). Honestly, I think the best answer here is tune the schedule - to make sure that users do not run backup jobs when they are actually working (i.e. events like lock or log off to triggers the backup). Another handy tool would be to disable backup over VPN/metered connections (you can possibility limit the WiFi networks as well).
For upcoming major version there would be a registry 'hack' to specify the exact adapters to use during the backup (and ignore others), that potentially can disable the backup over WiFi adapters.
Hope that helps!
-
- Lurker
- Posts: 2
- Liked: 1 time
- Joined: Jan 08, 2021 2:36 pm
- Full Name: Mike
- Contact:
Re: backup destroys network connection
Hello,
We're having a similar issue in our district. We have a combination of hardwired gigabit, wifi 400 megabit and wifi gigabit connections. We've enabled traffic throttling for the veeam project groups at different times to be 60-120 mb and still have each type of device lose internet connectivity during a backup. We can lower it further than 60 mb, but that's just far too slow of a backup process. We are backing up to NAS repositories on-site through our veeam proxy server. The NAS' are set up in Veeam B&R as SMB backup repositories, the proxy is set up as a Windows Repository. The servers and NAS' all use gigabit connections and the issue still occurs even if that's the only running veeam job.
Veeam support suggested looking at QOS settings on the firewall which we haven't gotten to yet, but reading the above thread makes it seem like this may be expected behavior? Before we start trying things out I'd like to try and find out more about whether our configuration is correct or if this is more or less the behavior we should expect. Is there anything we can try within veeam to keep that internet connection alive while the backup is running?
Thank you
We're having a similar issue in our district. We have a combination of hardwired gigabit, wifi 400 megabit and wifi gigabit connections. We've enabled traffic throttling for the veeam project groups at different times to be 60-120 mb and still have each type of device lose internet connectivity during a backup. We can lower it further than 60 mb, but that's just far too slow of a backup process. We are backing up to NAS repositories on-site through our veeam proxy server. The NAS' are set up in Veeam B&R as SMB backup repositories, the proxy is set up as a Windows Repository. The servers and NAS' all use gigabit connections and the issue still occurs even if that's the only running veeam job.
Veeam support suggested looking at QOS settings on the firewall which we haven't gotten to yet, but reading the above thread makes it seem like this may be expected behavior? Before we start trying things out I'd like to try and find out more about whether our configuration is correct or if this is more or less the behavior we should expect. Is there anything we can try within veeam to keep that internet connection alive while the backup is running?
Thank you
-
- Veeam Legend
- Posts: 934
- Liked: 214 times
- Joined: Jul 19, 2016 8:39 am
- Full Name: Michael
- Location: Rheintal, Austria
- Contact:
Re: backup destroys network connection
...that was also my thought above... I'm surprised that it's possible that a single process DoSes your local network connection... Is this really expected or maybe an design issue??
-
- Lurker
- Posts: 2
- Liked: 1 time
- Joined: Jan 08, 2021 2:36 pm
- Full Name: Mike
- Contact:
Re: backup destroys network connection
Update (potential solution): In our particular case, RAM usage seems to be the culprit. The Veeam agent was having issues connecting to the SQL LocalDB on the client machine. Logs and monitoring show an average of ~200MB of available RAM. By closing some unnecessary applications during the backup window and increasing the available RAM, we were able to have a week (and counting) of successful backups on the affected machine.
-
- Product Manager
- Posts: 2568
- Liked: 700 times
- Joined: Jun 14, 2013 9:30 am
- Full Name: Egor Yakovlev
- Location: Prague, Czech Republic
- Contact:
Re: backup destroys network connection
Thanks for update and sharing solution!
Who is online
Users browsing this forum: No registered users and 10 guests