Linux File Level Restore Speed

[tsightler]

So in testing a restore of a large file from Linux host we're seeing restore speeds of about 1MB/s. I'm restoring a single 3.5GB ISO file and it's going to take nearly an hour. Is this the typical expected speed for a file level restore? We can't really figure out what is taking so long, the VBK file is stored on a Linux host, but there is very little network activity, very little disk activity, and very little CPU usage, yet the restore is taking a very long time. Just trying to find out of this is a limit of the current technology.

Thanks,
Tom

[Vitaliy S.]

Tom,

Could you tell us where you're trying to restore that file? Do you have the same speeds while restoring small files?

[Gostev]

Tom, no this is unexpected speed - we would like to investigate this further. Besides what Vitaly has asked, for the sake of experiment could you please pull the VBK file to your Veeam Backup server, register the backup via "Import Backup", and perform the same operation locally?

[tsightler]

Just to a regular NTFS filesystem on an iSCSI disk mounted on the Veeam server. That disks is from an Equallogic iSCSI SAN and provides roughly 150-180MB/s performance with normal copies to and from the drive. Should have plenty of performance.

Restoring a directory of smaller files seems to offer similar speed, but generally a little better. Windows file restores from within the Veeam tool (rather than the File Restore Wizard) are basically the same speed. It appears that the Veeam file level restore driver may be limited to a very low queue depth, perhaps even 1 block at a time.

Thanks,
Tom

[tsightler]

Tom, no this is unexpected speed - we would like to investigate this further. Besides what Vitaly has asked, for the sake of experiment could you please pull the VBK file to your Veeam Backup server, register the backup via "Import Backup", and perform the same operation locally?

Unfortunately I couldn't do this because the backup in question was over 500GB and the local Veeam server simply doesn't have that much storage. So, to make things simpler (less variables) I decided to repeat the test with a Windows server that was only 36GB and test by restoring a file that was only 350MB. That eliminates the file level restore VM as the issue since restoring from a Windows can be done directly in the Veeam console. Here are the results:

Restore 350MB file from .vbk locate on Linux server -- Time: 4 minutes Speed: ~1.5MB/sec
Restore 350MB file from .vbk on local disk -- Time: 23 seconds Speed: ~15MB/sec

So basically, restoring the file from a local .vbk was approximately 10x faster. BTW, this is not a network performance issue between the Veeam server and the linux host as, when copying the VBK between the machines went about 60MB/sec. Something is obviously up when doing file level restores from a remote host. I'm going to try sharing out the .vbk with Samba to the Windows host and see how that performs.

[Gostev]

Tom, thank you very much for pinpointing the problem. I will ask QC to reproduce this and let development investigate. Thanks again! It would be interesting to see results for Samba test too, since in this case there will be no remote Linux agent used for transfer (which could potentially be causing this).

[tsightler]

Exported the same 36GB backup via Samba from the Linux server to the Veeam server, imported the backup, and restored the same 350MB file. Here are the results:

Restore 350MB file from .vbk via Samba share on Linux server -- Time: 12 seconds Speed: ~30MB/sec

In other words, it was actually faster from the Samba share than from local disk. I suspect this may be because, when restoring the local disk, both the read from the vbk and writes the restored file caused some contention.

So basically this problem definitely seem to be related to the remote host agent. That actually gives me a pretty reasonable workaround, if I know I need to restore a fairly large set of files just import the backup from the Samba share and restore from the imported backup. I plan to test this with the 500+ GB backup and the 3.5GB ISO tomorrow. Restoring that file took almost an hour today.

[Gostev]

Tom, thank you very much again for nailing down the issue.

[tsightler]

Any news on this issue at all? Were you able to reproduce the issue? Any chance of a fix? We've been OK with running our backups to an iSCSI attached volume, but our life would really be easier if we could push our VBK files to our Linux server that's directly attached to our tape library which was our original plan.

[Gostev]

Yes, issue was easily reproducible and according to the records it is fixed in 4.0 code.
I'll find out if it is not too late for the fix to be added to 3.1.1 maintenance release.

[Gostev]

Tom, the 3.1.1 code is in final testing right now, so we would like to avoid changing it.
How about if we provide you a separate fix for this issue that you can apply on top of 3.1.1?

[tsightler]

A separate fix would be fine if that's something you can do with reasonable effort. Otherwise I'll just "tough it out" with the current limitation until 4.x get's released. It's not a major problem for us, more of an annoyance than anything. If this is something you can do easily, well, I won't turn it away.

[Gostev]

OK, the fix appears really easy - just about changing the I/O buffer size in a couple of places in the agent code increases the FLR speed 10-15 times, but we would like to spend some time to make sure this does not affect some other functionality except FLR in some bad way, because the agent does a lot of other things too.

[tsightler]

Hate to bring up an old thread, but with Veeam 4 I'm still experiencing this problem, but only in some cases. We push all of our backups to a Linux target, and full system restores are reasonably fast, as well as file level restores from directly withing the Veeam GUI. However, we're still seeing poor performance when performing a restore using the FLR appliance when the VBK file is on a Linux host. In other words here's what we seem to have now:

1. Veeam GUI FLR, VBK on Windows local disk -- FAST
2. Veeam GUI FLR, VBK on Linux remote host -- FAST
3. Veeam FLR Wizard, VBK on Windows local disk -- FAST
4. Veeam FLR Wizard, VBK on Linux remote host -- SLOW

Any chance that the "fix" for this issue somehow didn't make it into the Veeam FLR appliance agent?

[Gostev]

I will check with the devs.