Feature Request - improve network performance for linux agent backup

[ebondi]

Hi, I need backup a huge amount of data to AWS S3
Original data are on a Linux host with Veeam Agent for Linux.

I already spoken to Veeam Support (case 06091179) that suggested me to create custome registry key S3ConcurrentTaskLimit on Veeam Repository servers and Veeam Server (all Windows machines). Well, that custom parameter works well and is able to create many more multiple network connection from Windows servers to AWS S3 web sites for other kind of jobs, but the backup job that read data on Linux host doesn't inherit that parameter so a maximum of 64 network connections (the default) cannot be incresed to let's say 128.

Veeam Support told me that currently there is no custom parameter S3ConcurrentTaskLimit for Veeam Agent for Linux and I don't understand why. I simply need it otherwise the backup window is not met due to the huge amount of data to transfer on cloud repository.

Please help and simply include that feature also on the Linux version of Veeam Agent!

Thanks
Enrico

[HannesK]

Hello,
and welcome to the forums.

Is my understanding of the case correct, that you are using file-based backup instead of block-based backup? If yes, what is the reason for file-based backup and what performance do you get with block-based backup (volume level / entire machine backup)?

Asking for faster file-based backup is valid, but I like to emphasize, that the agent is built for block-based backup. For scalable file-based backup there is the NAS backup functionality available.

Best regards,
Hannes

[ebondi]

I use file level backup without file system snapshot because the linux host has ZFS file system.
I already tested NAs backup and this is slower too.

I think I already tested all possible Veeam supported solutions for this kind of file system and my final goal now is just improve throughput bandwidth usage so I just need the custom S3ConcurrentTaskLimit for Veeam Agent for Linux.

Is just possibile have this like the Windows version?

[HannesK]

I already tested NAs backup and this is slower too.

slower than what? And can you maybe tell us what "huge amount" means? 100TB, 1PB, 10PB? Do you maybe have a case for the NAS backup performance issue? Is it about incremental performance or full backup performance?

S3ConcurrentTaskLimit would probably not help. There are other optimizations that need to be done before that would help. The point is, that NAS backup should be fast (assuming it's not dozens of millions of 1KB files... then every file-based backup is slow)

[ebondi]

NAS backup has also a problem when I make a restore because it write restored file all with root permission and not writing the original user and group id, so that method is not good at all.
NAS backup to S3 goes maximum at 60 MB/s, the same speed reached by backup via Linux Agent, so the limit seems related to transfer to AWS S3 repository.

When I say high amount of data I mean currently the file server is around 400 TB of data.

In every case I speak about performances I mean during full backup.

I found that during a Full backup for a directory of the linux file server I see the single job for Veeam Agent based backup creates 70 parallel access on the file systems files (checked by lsof command) and maximum 64 network connection to AWS S3 web server (check by netstat command)

Veeam Agent based backup are limite to run 1 job at a time by design so I cannot create multiple jobs to run at same time in order to try parallelize network connections to S3, so I tryed the following workaround.

I created a secondary Linux file server that mounts the same file systems of the first linux server, then I setup another Veeam Agent based backup job for the secondary linux server. Then I started the backup for Linux server 1 directory 1, and at the same time I started the backup for Linux server 2 directory 2 (just to say it is a different one, not the same data of the first job) and I see that server 2 can read data from the first linux server via NFS with no performance degradations and then transfer backup to AWS S3 basically doubling bandwidth usage to AWS S3 web site.

My test says that:
I need use Veeam Agent based backup job because NAS backup is not good enough in case of restore, and that I need Veeam Agent make more connection to AWS S3 because my file server infrastructure as no bottleneck performance issues (disk or bandwidth) and that Veeam can do MORE by design in order to speedup transfer.

Hope to be clear with descriptions of my tests

[ebondi]

Just to remind the Veeam Agent performance, at this time I've launched a backup job on the Windows server that has the Veeam repository and it is uploading at 1900 Mbps with S3 custom setting to 128 parallel connection (default is 64) while the backup job on the Linux server is performing at 850 Mbps (with its embedded default 64 connections). It doesn't seem a case that Windows server is upload 2x than Linux server...

I repeat: what is the difficult to introduce the same S3 custom connections setting on the Veeam Agent for Linux as already done for the Windows version?
Why Veeam invented the S3 custom setting for the Windows version? I'm sure someone asked for it for the same purpose to improve performance. So why don't create that custom setting for the Linux version?

Solution could be very easy and close.

[HannesK]

NAS backup has also a problem when I make a restore because it write restored file all with root permission and not writing the original user and group id, so that method is not good at all.

Do you maybe have a case number for that? NAS backup restores permissions / ownerships correctly. If not, then it's a bug.

NAS backup to S3 goes maximum at 60 MB/s, the same speed reached by backup via Linux Agent,

thanks, that's good information!

Just to remind the Veeam Agent performance, at this time I've launched a backup job on the Windows server that has the Veeam repository and it is uploading at 1900 Mbps

just to be sure... that is Veeam Agent for Windows backing up to a Windows Repository and then uploading to S3? That Veeam Agent for Windows job is file-based or block-based (volume / entire machine backup)

Agree, from what you describe it seems that more connections might help. Let me check.

[HannesK]

Hello,
can you try the following?

create a new file /etc/VeeamAgentConfig and add the “S3ConcurrentTaskLimit” with a desired value. For example

Code: Select all

 S3ConcurrentTaskLimit = 128

In the logs, you should then find a log entry that says "Non-default option 'S3ConcurrentTaskLimit'". Let us know, how it goes.

Best regards,
Hannes

[ebondi]

Hi, that file worked! Now agent can create the number I want of multiple network connections to AWS S3 !

Now you should explain why official Veeam Support didn't told me that custom setting and in which way you know that custom setting. Do you work at Veeam company?

[HannesK]

Hello,
how fast is it now?

Because support did not escalate to R&D. That option is not described in any public or support documentation. Developers did not expect, that it might be useful, so the documentation did not exist. Humans, you know

Best regards,
Hannes