Major Flaws in OS File Restore for Linux

[kurt2439]

I hope other people can comment on these issues as I think they are important ones for Linux users of Veeam Backup and Replication. Hopefully this thread will bring improvements to light if the community sees them as important

I have worked with tech support up to tier 3 and I have confirmed that in the OS file restore there are the following issues. They are not likely to be resolved unless there is significant user moment behind them:

1a. Guest OS file restore is VERY slow when restoring multiple files/folders. A single file restore will be quick but if the restore system needs to enumerate multiple files from the backup then restore speeds are typically around 50kbps. For us, this means that a 2GB restore of a user folder in Linux took 2.5 hours. This is not acceptable for a enterprise backup system.

1b. Because Veeam is aware of the poor performance of the FLR utility for GUI restores, they enable FTP in the FLR. However it is not possible to preserve permissions when transferring the files via FTP. Therefore there is currently no way to preserve linux file permissions on a guest os file restore, and even when they fix issue 3a, the restore speed will be so slow as to only be useful for extremely small restores.

2. The FLR doesn't support Linux ACL's. Linux ACL's are not just the owner:group:other permissions. Linux ACL's are set via 'setfacl'. These permissions can be extensive and not restoring them is a serious limitation of the product. The FLR does not mount ext3 volumes with the 'acl' option but that would be a start. Read more at http://www.centos.org/docs/5/html/Deplo ... -acls.html.

3a. Using the "Preserve Permissions" option on the Linux Guest OS files restore doesn't work. Only the file owner is restored. Group is never restored. This is a known issue that will be fixed.

3b. The base file permissions of the folder are not restored at all. It's not clear if they intend to address this but they should. Otherwise in order to maintain the permissions of a restore, you need to restore from one folder level back which could mean an incredible addition of restore data.

The point of my post is not to bash Veeam, I think the product works wonderfully for backing up and restoring full VMs and VM files. I'm also aware that there are *workarounds* for some of the issues above like restoring a virtual disk and hot adding the disk or running an instant VM and rsync'ing the files to the production system. I'm also not taking out anger on any Veeam technical staff, I was encouraged by multiple people to try to stir up discussion on the forums about these topics to make these issues a priority.

My point here is that these issues exist and should be fixed to make restores as painless as possible. No one wants to deal with a workaround when they need to restore data. Also some of these issues run counter to the claims of the product, preserving linux file permissions for instance, which is mentioned several times in the user guide and other documents.

[tsightler]

I agree that it would be nice to see some improvement in Linux FLR, however, I would like to point out a few inaccuracies in your current statements as well as to provide some possible alternatives. I recognize that none of these are ideal, but sometimes all we can shoot for is "good enough" while making do with the current situation.

1b. Because Veeam is aware of the poor performance of the FLR utility for GUI restores, they enable FTP in the FLR. However it is not possible to preserve permissions when transferring the files via FTP. Therefore there is currently no way to preserve linux file permissions on a guest os file restore, and even when they fix issue 3a, the restore speed will be so slow as to only be useful for extremely small restores.

This particular statement is not correct. There are many FTP clients that support preserving file permissions, including what is probably the most popular command line Linux ftp client "lftp". Using "lftp" with the mirror option is quite easy and generally reasonably fast and it will preserve permissions. Of course, you can also use SFTP or even netcat to pipe the files via tar. See FLR Multi-OS via FTP no files with dots for more info.

Regarding ACL's, I agree that it would be nice to have this feature added, however, for now there are many options. For example, you can use instant restore to bring up the VM in a virtual lab and then copy the files using native tools that support ACLs, however, probably the easiest option is to simply backup your ACLs with the native tools and then restore them after performing the file level restore.

For example, "getfacl -R somedir > acls.txt" will backup a set of ACLs, and "setfacl -R --set-file=acls.txt somedir" will restore those ACLs. Another nice option is a tool like ACLbit http://aclbit.sourceforge.net/ to backup and restore ACLs.

Once again, I'm not saying I disagree with you, just offering some options to mitigate the current shortcomings. In my previous role I made heavy use of Veeam in a Linux environment and these are the methods that we used with generally good success.

[kurt2439]

I agree about functionality of the workarounds for ACL backup and restore but it's a needless pain as it should work as they say it does out of the box. And some people will assume this works as they say and then panic when they find out it doesn't.

I figured our FTP restores weren't preserving permissions due to some setting in our client but the tier 3 tech said that permissions won't restore via FTP so I just took him at his word. One of the tier 2 techs said that in order to preserve permissions via FTP the files must be zipped on the FLR first, but the FLR volumes are mounted read only. I didn't try remounting.

If the FLR supported rsync and mounted with acl support, that would be the least painful option I would think.

[tsightler]

Yes, rsync with ACL support would be nice. Back in the 4.x versions there wasn't even an FTP option and I asked for rsync then, but I think FTP was easier to implement as that was already a part of the Busybox build on the ISO, so I understood and accepted FTP as better than nothing. Improvements are always possible in the future and it customer feedback that makes it happen so thanks for that,

[kurt2439]

Are there any other linux backup admins out there that find the lack of ease and speed in guest os file restores troublesome? Or worried about the difficulty in preserving permissions?

Or are others only using this as a virtual machine backup tool while relying on a second backup strategy for file level restores?

[cdickerson]

Wow, great timing. I too was going through these exact same exercises this past week with Veeam support and came to the exact same conclusions and answers from Veeam. the other missing feature is symbolic links. when you restore individual files, the symbolic links are lost for that file. I didn't try restoring a large folder to see what type of speed I was getting. I'm actually looking at the new Backup Exec 2012 with the Linux Agent as an alternative.

-Craig

[jvandevelde]

I have had the same type of problem in April last year on the 5.0.2 release.

This was logged by support under case #5126463.

The workaround I was given was to use tar on the VM helper system to pipe the files to the target system.
e.g. tar cf - . |ssh 10.10.x.y "mkdir /tmp/restored_using_veeam.20110408;cd /tmp/restored_using_veeam.20110408; tar xf -"

In order to be able to do this, you have to have the secret root password for the VM helper system.

The final status was that 'the issue will be logged on the feature request list'.

[kurt2439]

tar cf - . |ssh 10.10.x.y "mkdir /tmp/restored_using_veeam.20110408;cd /tmp/restored_using_veeam.20110408; tar xf -"

That's a clever workaround assuming it works. I'll have to try it out. I don't think the password is published but the user 'root' and the password is '[Veeam_Server_Name]_r' where the Veeam_Server_Name is case sensitive.

That's a strange issue about the symlinks. I would think it would still work as long as the file name and path is the same.

I really encourage anyone who reads this post who is affected by this issue to post and at least let the Veeam Staff know that you are affected and would like to see a resolution. The sales guy I spoke to said he would forward along the issues to his manager but I think the only way these are like to be resolved in any timely manner is if we show there are many people affected by this who would like to see a resolution.

[tsightler]

It works. I actually posted a link with this same method in my original reply, actually, it even has a faster option, using netcat instead of SSH so that you don't pay the encryption penalty.

[astrium-fdh]

That's a clever workaround assuming it works. I'll have to try it out. I don't think the password is published but the user 'root' and the password is '[Veeam_Server_Name]_r' where the Veeam_Server_Name is case sensitive.

i copy a rsync binary to the helper system and then use rsync, i think it's a little bit easier than piping with tar.
i hope the file permissions issue will be fixed soon

Lutz