Linux agent restore Performance

[Qasim]

Support case ID: #05547077

Background: We have deployed managed Veeam agent for Linux to backup the ZFS file system. VAL is targeting the Network share mount on the Linux box and backing up in Snapshot-less mode. VAL directly backup to the VBR local repository and then Backup copy jobs copy the Backup to cloud connect. Backup copy job also configured with GFS. Most of the backup size around 15 TB to 20 TB.

Issue:

When we try to restore from the Cloud connect repository. There is no restore option available from VBR. As these are the VAL file level backup so as a work around we import the backups to a stand alone VAL. Once the backup are imported it directly mount from the Cloud connect to the Linux box and then we can run copy command to copy the mounted data to local storage. But the issue the performance is very slow. We did speed test which results Linux box has download speed up to 400mpbs . Also Iperf test between Linux box and cloud gateway shows data transfer of average 114 mbps and 200mbps max. While running the restore we have monitor the network and it shows that VAL restore is not using bandwidth effectively and it is limited to 30 to 60 mbps. One of the reason that VAL cannot utilize the available bandwidth is it can do only one thread download.

Feature Requeest:
(1)VBR File level restore for VAL agent backup saved on Cloud connect
(2) Multithread download for VAL restore

[HannesK]

Hello,
normally I would say, that NAS backup should solve the issue (VAL is build for block-based backup, not for file-based / NAS backup). But NAS backup does not support cloud connect today. NAS backup will support backup to object storage in V12, just in case that is an alternative for you.

Overall: your feature requests are valid and we have them on our list. I add your vote +1

Best regards,
Hannes

[MichailGalushkin]

in other words, is it normal that the recovery speed is about 30 megabits? How to use it in this case?

[MichailGalushkin]

I have a same problem, 11a + last linux agent on Centos 7 x64
When i try to copy some file from monted directory /backups i getting only 3-5 megabyte per second on gigabit link. But in same time i can start job to backup i'am getting 400-500 mbps.

[HannesK]

Hello,
and welcome to the forums.

No, it's not normal to have only 30mbit/s restore speed. It depends on file size and hardware (millions of very small files are slower than a few large files).

When i try to copy some file from monted directory /backups i getting only 3-5 megabyte per second on gigabit link

what is your backup target? What speed do you get, if you do a file copy test from that target with the same data?

But in same time i can start job to backup i'am getting 400-500 mbps.

which is slower than 1Gbit/s, so I guess the backup target is slow (IO performance). For backup speed: if you use the default settings, then backup is done block-based. That's usually faster than file-based backup. File level restore is file-based. That's usually slower than block-based restore. Block-based restore only works for volume-based restore.

Overall, I guess the reason is small files combined with a slow backup target. If my guess is wrong, please double check with support and post the case number for reference.

Best regards,
Hannes

[MichailGalushkin]

Hello!
Thank you for the reply!

what is your backup target? What speed do you get, if you do a file copy test from that target with the same data?

Target speed if i copy the same file(one big file 3gb) is around 50-55 mbps (1gbs link), if i use agent as standalone and make a same backup(using NFS target location) to same server with NFS role installed i'am getting close speed's for backups and restore, but the problem is when i use backup to repository, if I choose Veeam Backup & Replication repository speeds for backups is around 70-100 megabyte per second, but if i want to restore this file i'am getting only 30 mbps(3-3,5 megabyte per second).
Backup test server is HP GEN9 with NVME 1,6TB source server is also has a NVME disk, so it's clearly not the disk speed that's the problem here.

REFS information from backup server
REFS Version : 3.4
Number Sectors : 0x00000015b3c00000
Total Clusters : 0x000000002b678000
Free Clusters : 0x000000002b176928
Total Reserved : 0x00000000002c7a8d
Bytes Per Sector : 512
Bytes Per Physical Sector : 4096
Bytes Per Cluster : 65536
Checksum Type: CHECKSUM_TYPE_NONE

[HannesK]

Hello,
hmm 50-55mbit/s vs. 30mbit/s is all slow for 1gbit/s network. Overall, my understanding is, that the fastest file copy speed you see is 50-55mbit/s. Is that "to the backup target" or "from the backup target to the VAL machine"?

Backup is block level per default, so that's irrelevant (I did not read anything else, so I assume block level backup. that is volume level or entire machine backup)

If it's slower from a Veeam repository, then the repository server might have some other hardware constraints. Support can help to figure it out.

Best regards,
Hannes

[MichailGalushkin]

hmm 50-55mbit/s vs. 30mbit/s is all slow for 1gbit/s network.

Sorry of course 50- 55 megabyte not a megabit per second (1gbps)

[MichailGalushkin]

Server information: Build 11.0.1.1261 P20220302
Agent information: veeam-5.0.2.4567-1.el7.x86_64.rpm
KMOD: kmod-veeamsnap-5.0.2.4567-1.el7.x86_64.rpm
Linux(source): Linux 3.10.0-1160.62.1.el7.x86_64

[HannesK]

no problem. I still don't see 100% that you compare apples with apples (you might, but I cannot see it written clearly).

The test must be the same files native and in Veeam backups. E.g. one big file vs. one big file. And the copy direction must be from backup storage to the machine where the agent is running in both cases.

if there are significant differences, please check with Veeam support and post the support case number for reference

[gtelnet]

This sounds similar to a post I wrote in April of 2021, where you can see my workaround. It's not ideal, as it creates more manual steps, but it does get the restore completed 10x faster.

veeam-agents-for-linux-mac-aix-solaris- ... 73502.html

[MichailGalushkin]

Yes, that's exactly what I'm talking about. For some reason, dear HannesK pretends that he does not understand what I am talking about. How can I restore a 10 gigabyte file in a reasonable time if the copy speed from the repository is 35 megabits, yes i know that i can use NFS as a repo and in this case i'am getting full gigabit speed for backup and restore, but...
Is it really just the two of us experiencing such problems?

[HannesK]

Hello,
what Greg describes is the lack of async read and multi-threading, yes.

Michail: I just wanted to be 100% sure, that it's not the classic mistake I have seen very often in the past, that somebody compares the speed of one multi-gbyte file with thousands of small files.

overall, I can only repeat that we have plans to improve restore speed. If you already checked with support that there is no misconfiguration, then there is nothing I can add right now.

Best regards,
Hannes

[MichailGalushkin]

Hannes, no it's only one 5gb file, not a "thousands of small files"
If i right understand, it's not possible to get support to help me with a community edition?

[HannesK]

you can try, but chances are low to get support on "best effort" basis.