Worker Instances - Problem with available IPv4 addresses in the subnet

[vincenteberwein]

Hello,

As I read in the documentation, when performing backups for each EC2 instance specified within a policy, exactly one worker instance is also deployed. If I am correct, this means that within a subnet there must always be at least the same number of IPv4 addresses available as there are EC2 instances to be able to successfully back up all instances. We currently have the problem that our subnet does not have again the same number of free IPv4 addresses as there are EC2 instances in the subnet. This causes an error in our backup policy and Veeam Backup for AWS reports that there are not enough free IPv4 addresses left to perform the backup for certain EC2 instances.

Now my question: what is the best practice in this case? Do we need to configure our subnets so that there are always at least as many free IPv4 addresses as there are EC2 instances in the subnet or is there also a way to configure Veeam Backup for AWS so that one worker instance is not created per EC2 instance to perform the backup, which means that one worker instance can back up multiple EC2 instances? The latter approach would be much more efficient.

Kind regards
Vincent

[nielsengelen]

Hi Vincent,

A worker will still be needed per EC2 instance, currently - we don't support multiple EC2 instances via a worker. I'll consider this as a feature request for an enhancement.

Normally, you configure an additional subnet that can be used for the worker assuring you have enough addresses. We have a soft limit built-in right now to deploy a maximum of 20 workers at the same time. Am I correct in stating you are using somewhere around 230-240 addresses all the time?

[vincenteberwein]

Hello Nils,

Thank you for your reply and for considering our concern as a feature request.

Okay, that with the soft limit is good to know. Correct, there are currently only about 25-30 IPv4 addresses available in our subnet. However, despite the soft limit, this number of IPv4 addresses is not enough because AWS probably does not release IPv4 addresses as quickly when a worker instance is terminated and therefore we still exceed the limit of IPv4 addresses.

I think we also need to set up an additional subnet for the worker instances to solve the problem. Or do you see another possibility?

Kind regards,
Vincent

[vincenteberwein]

Hello Nils,

I checked one of our backup policies again last night and monitored the deployment behavior of the worker instances. Unfortunately the soft limit of 20 worker instances did not work, so at peak times more than 30 worker instances were deployed in the same subnet at the same time and were also running in parallel. This resulted in our available IPv4 addresses being fully utilized.

Kind regards,
Vincent

[nielsengelen]

Hi Vincent,

Let me do some verification internally. In the mean time, could you maybe open a support case for additional insight via logs and maybe an extra pair of eyes? I think it may work faster that way with combined efforts. Can you let me know the support case ID as well so I can use it as a reference?

[vincenteberwein]

Hello Niels,

thank you for taking care of this. We have opened a support case in the meantime. The support case ID is 05127151.

Kind regards,
Vincent

[vincenteberwein]

Hello Niels,

just for your information. Support told us to connect to the Veeam Backup for AWS appliance via SSH. There we should edit the config.ini file located at /etc/veeam/awsbackup/ and add the following option:

[COptions]
MaxParallelTasks=25

After that the machine should be rebooted.

With this option you could limit the maximum parallel active worker instances to 25.

We did this, but unfortunately over 30 parallel active worker instances were started again.

Kind regards,
Vincent

[nielsengelen]

Hi Vincent,

Please keep working with support for now so they can analyze why the limit is not being maintained.

[nielsengelen]

Hi Vincent,

I noticed that your case has been resolved together with some other questions. Please let me know if you have any other questions ongoing.

[vincenteberwein]

Hello Niels,

yes, we were able to solve the problem successfully.

Currently, I have no further questions in this regard.

Thank you.

Kind regards,
Vincent