Agent-based backup of Windows, Linux, Max, AIX and Solaris machines.
Post Reply
hpadm
Enthusiast
Posts: 51
Liked: 10 times
Joined: May 18, 2021 1:55 pm
Location: Slovakia
Contact:

Slow individual file restore speed but 1Gbit/s network traffic

Post by hpadm »

I have a monthly full backup (275G / 800G) and 15 incremental backups (~1G / 2.5G). I have started an individual guest file restore job on several database files, ~40G uncompressed. The job is progressing at 8 MB/s (4 files concurrently at 2MB/s each) and will take 1.5 hours. Meanwhile, the gigabit downlink from the backup NAS is staying fully saturated the entire time, and will receive ~640G of network data total.

It seems as if the restore process is downloading the entire backup image, several times over, and is slowly picking out the relevant data blocks. The NAS gateway, backup proxy, mount server, etc are all on the same bare metal host, so the data path should be as efficient as possible. I have also enabled guest file indexing several months ago, thinking that maybe a missing index was causing the restore process to be very inefficient. But that doesn't seem to be the case.

I would like to ask, is it supposed to be like this? Downloading 640GB of data to restore 40GB? Or do I have a severe misconfiguration somewhere?
PetrM
Veeam Software
Posts: 3229
Liked: 520 times
Joined: Aug 28, 2013 8:23 am
Full Name: Petr Makarov
Location: Prague, Czech Republic
Contact:

Re: Slow individual file restore speed but 1Gbit/s network traffic

Post by PetrM »

Hello,

May I ask you to open a support request and share its ID for our reference? I also asked my colleagues from QA department to research this scenario deeper in their environment.

Thanks!
hpadm
Enthusiast
Posts: 51
Liked: 10 times
Joined: May 18, 2021 1:55 pm
Location: Slovakia
Contact:

Re: Slow individual file restore speed but 1Gbit/s network traffic

Post by hpadm »

I have submitted it as Case #05635057.

I have run multiple controlled tests. Both our bare metal windows hosts and hyper-v servers can reproduce this. It also happens when restoring directly from a full backup.

The only anomaly is that when I perform restore to the main server running VBR Console, via 'Copy To', then the restore happens much faster and with high network efficiency, as the target host's agent obtains exactly the data it needs and at near gigabit speeds, pushing it to the VBR Console host as it goes. Reported performance was 60MB/s while Task Manager showed around 400Mbit/s on the VBR Console host. I have no idea why this roundabout method works efficiently while the default direct one is so bad.
Dima P.
Product Manager
Posts: 14396
Liked: 1568 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: Slow individual file restore speed but 1Gbit/s network traffic

Post by Dima P. » 1 person likes this post

hpadm,

Try to start File Level Recovery on the machine, then instead of using Backup Browser navigate to C:\VeeamFLR folder and copy the needed files to the file system. Please let us know if it runs faster or not. Thanks!
hpadm
Enthusiast
Posts: 51
Liked: 10 times
Joined: May 18, 2021 1:55 pm
Location: Slovakia
Contact:

Re: Slow individual file restore speed but 1Gbit/s network traffic

Post by hpadm » 1 person likes this post

That is a smart thing to test! When I did that directly on the restore target (which its its own backup gateway), I got 640 MBit/s. When I used the 'Mount to console' button and formed a 2-hop circuit, I saw around 300 Mbit/s (I guess at these speeds, any added bit of processing/relaying delay slows things down). So if I obtain the data directly through the mount points, the issue is not there!
Dima P.
Product Manager
Posts: 14396
Liked: 1568 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: Slow individual file restore speed but 1Gbit/s network traffic

Post by Dima P. »

hpadm,

Glad it worked! Looks like the performance decrease in case of FLR is related to multiple system checks that are performed within operating system while when you do a copy from the folder all checks are handled within the OS itself. We will discuss with RnD folks if it's possible to optimize current FLR performance.
vmtech123
Veeam Legend
Posts: 235
Liked: 133 times
Joined: Mar 28, 2019 2:01 pm
Full Name: SP
Contact:

Re: Slow individual file restore speed but 1Gbit/s network traffic

Post by vmtech123 »

Are you running Enterprise Manager? If you are not, I'd read up on how indexing works with Veeam.
hpadm
Enthusiast
Posts: 51
Liked: 10 times
Joined: May 18, 2021 1:55 pm
Location: Slovakia
Contact:

Re: Slow individual file restore speed but 1Gbit/s network traffic

Post by hpadm »

I really don't know what's going on or why the FLR operation is behaving like this. The followup tests that you and the support staff suggested, show even more that the operation is severely malfunctioning. I was expecting a simple answer to a trivial configuration problem, but it seems this is an anomaly that has not been previously observed.
  • The NAS shares and backup operations are able to run at near-gigabit speeds.
  • Directly copying files from FLR mount points is able to run at 640 Mbit/s.
  • A file index is present. However, this same behavior was observed prior to adding the index. According to documentation, this index is only used for searching by filename, and for "1-click restore in VEM", whatever that is.
  • The backup should know where each file's blocks are located. There should be no need to read and decompress the entire backup from start to end or whatever.
  • During restore, the file's blocks are gradually loaded in, at a fairly slow rate, throughout the whole operation. I assume that the file is stored consecutively in the backup, so if any data seeking was required, I would expect a long period of no progress, followed by a quick burst.
hpadm
Enthusiast
Posts: 51
Liked: 10 times
Joined: May 18, 2021 1:55 pm
Location: Slovakia
Contact:

Re: Slow individual file restore speed but 1Gbit/s network traffic

Post by hpadm » 2 people like this post

After extensive testing, the issue was reproduced and identified. Veeam on windows during FLR was using a 4MB read-ahead block size even though the backup's blocks were 256kB. I assume this meant that it was re-reading the same parts over and over while only keeping a small portion. 4096/256 = 16x, which matches the measured overhead during testing. It was found to only happen with a windows backup gateway, probably only during FLR (full VM restore runs fine), only for NFSv4 (SMB and NFSv3 run fine), and is mitigated with a lower deduplication setting. A fix will be included in the next major update.
Dima P.
Product Manager
Posts: 14396
Liked: 1568 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: Slow individual file restore speed but 1Gbit/s network traffic

Post by Dima P. »

hpadm,

Great result! Big thank you for your help with the investigation!
Post Reply

Who is online

Users browsing this forum: No registered users and 8 guests