UDP traffic seen as DDoS attacks

[dood]

Hi'

We're actually using Veeam (9.5 last update) to backup external customer's servers to our dedicated Veeam server installed in a datacenter.

Customers use their fiber WAN link, we mount IPsec VPN between customers and our server.
Each customer has a local Veeam proxy. For some of them we only do copy job.

Our datacenter offers an Arbor system (DDoS protection), In fact this is mandatory for everyone.
We have a lot of difficulties to do not trigger this protection system with our backups.
As soon as we start a backup, like 10 min later, Arbor sees an UDP attack and mitigates/blocks traffic.

We discuss a lot with the datacenter team but they do not seem to be able to make exception on our customers public IP.
They try to make exception on ports but the range used is too big for them (2500-5000 UDP).

So we are a little bit stuck for the moment.

Did you guys encounter similar cases, how did you handle it ?

Could we for example restrict this UDP range for something like 2500 to 2510 ? (we don't have a lot of jobs)
I probably already know the answer but whatever, is their a way to tell Veeam to use TCP traffic instead of UDP ? ... yes I know ... I'm desperate
Any other idea will be appreciated

[Mike Resseler]

Hi,

I am a bit confused about this. You are protecting servers of customers, and you are using your own Veeam server. I assume that the customers don't have a local Veeam server?.
You are using IPsec VPN between your site and the different customers. So am I right in assuming that it means that the backup of the customers servers goes over private IP instead of public IP? (You talk about UDP traffic on the public IP but VBR should not make a connection to the public IP)

Second: Each customer has a local proxy. Which means we will see traffic from the local proxy to VBR server and to the VBR repository.

What type of setup is there exactly? Customer has VMware or Hyper-V? Local Proxy (Windows), Repository server is what type of server/ connection? And are you using a WAN accelerator?

(Sorry for all the questions)

Cheers
Mike

[dood]

I am a bit confused about this. You are protecting servers of customers, and you are using your own Veeam server. I assume that the customers don't have a local Veeam server?

You're right, customers generally get a simple ESXI installed on a physical server with one or two VM.
One (Windows) VM acts like a Veeam Proxy.

You are using IPsec VPN between your site and the different customers. So am I right in assuming that it means that the backup of the customers servers goes over private IP instead of public IP? (You talk about UDP traffic on the public IP but VBR should not make a connection to the public IP)

Yes, there is an IPsec VPN between our site and each customer.
So yes, the Veeam Proxy server of the customer contacts our Veeam server through his VPN private IP.

Second: Each customer has a local proxy. Which means we will see traffic from the local proxy to VBR server and to the VBR repository.

Exactly

What type of setup is there exactly? Customer has VMware or Hyper-V? Local Proxy (Windows), Repository server is what type of server/ connection? And are you using a WAN accelerator?

Customers have VMware, Windows local proxy.
Our repository server is a Windows Server 2016 with an attached ISCSI storage.
We don't use WAN accelerator.

(Sorry for all the questions)

No problem

[Mike Resseler]

What I am guessing here is the connection from the proxy/repository to the proxy (a windows server) which uses TCP 2500 to 5000. You can see it here in the list: https://helpcenter.veeam.com/docs/backu ... tml?ver=95

The thing is, I don't see UDP in that list.

What you could do is go to the backup infrastructure, select managed servers/ windows servers. Select the correct server and do edit. At the second page of the wizard you can change the ports range. (You will see 2500 to 5000 there). And see if that can help you. Important to note is that with less ports, you can get notifications when a job runs that Veeam can't find a free port.

I would start there and look what the network is doing

My thoughts
Mike

[dood]

What I am guessing here is the connection from the proxy/repository to the proxy (a windows server) which uses TCP 2500 to 5000. You can see it here in the list: https://helpcenter.veeam.com/docs/backu ... tml?ver=95
The thing is, I don't see UDP in that list.

Damn, you're right, I saw 2500-5000 UDP but my mistake !

The Arbor protection trigger on UDP traffic (see as flood I think), so it's probably due to :
Shared folder CIFS (SMB) share ; TCP/UDP ; 135, 137 to 139, 445 ; Ports used as a transmission channel from a backup proxy to the target CIFS (SMB) share.

What you could do is go to the backup infrastructure, select managed servers/ windows servers. Select the correct server and do edit. At the second page of the wizard you can change the ports range. (You will see 2500 to 5000 there). And see if that can help you. Important to note is that with less ports, you can get notifications when a job runs that Veeam can't find a free port.

As you said, this traffic is TCP so Arbor doesn't care about this. I need to focus on UDP traffic (Shared CIFS).
Any way to avoid UDP traffic in data transfer ?

EDIT : or maybe it's the encapsulated IPsec traffic (which is UDP) ...

[Mike Resseler]

I don't believe that you can avoid that UDP traffic unless you work with a gateway. In that case you would have a backup proxy (tenant) to a gateway server (at your side) and then from the gateway server to the SMB share.

But seeing your edit... I might seek in that direction before making major changes to your setup

[dood]

As a temporary solution, we found that if we run jobs one by one, Arbor Protection is not triggered.
So as soon as Arbor see 2 backups (or probably anything else with a lot of UDP traffic) from 2 differents public IP it considers as flood and proceed to mitigation.

We're currently asking to add exceptions on public IP/ports ...

I'll keep you informed !

[yasuda]

IPSEC will work over TCP, but there might be a performance loss, depending on connection quality.

If you use TCP and a packet is lost, there's a dual re-transmit request: The IPSEC connection and the TCP connection inside the tunnel.

If you run IPSEC UDP and a packet is dropped, the connection inside the tunnel will request re-transmission. IPSEC doesn't need to worry about dropped packets. And if you're running UDP in the tunnel, you really don't want IPSEC re-transmitting dropped packets.

[mcz]

IPSEC will work over TCP, but there might be a performance loss, depending on connection quality.

IPSEC could also be tunneled via UDP...

[dood]

Got some news here.

Datacenter guys explain me this about exceptions on Arbor :
- Arbor will be triggered by different behaviours (like UDP flood, fragmented packets, ...), this CAN'T be changed. This is the same for every one.

But you can add filter to bypass mitigation when Arbor is triggered. You can whitelist for example your headquarter public IPs.
In my personnal case, customers can connect from anywhere so I need to whitelist all public IP all over the world

We actually run jobs one by one and it's working fine (but a little bit touchy as we have to make sure that there's only one job running at time / we have copyjobs run on customer side).