Backup to tape of big amount of files

[diputaciogirona]

Good morning everyone,

Currently, for our CIFS and NFS resources, we perform backups in three steps: backup to disk, backup copy to disk, and then tape backup using the NDMP protocol.

We are backing up volumes with more than 25 million files, and this is the only method we’ve found that allows us to keep the backups within the available window without causing the repository database to grow excessively. With NDMP, Veeam does not store the list of files inside the volumes—it backs up the entire volume as a whole.

Soon we will be changing our storage system and will no longer be able to use NDMP, which means we’ll have to switch to either “file to tape” or “backup to tape.”

We have tested “backup to tape” using a backup copy chain as source and “file to tape” using SMB or NFS as source. In both cases, Veeam stores the complete list of files copied to tape in the tape.files and tape.file_versions tables of the PostgreSQL database. As a result, backup jobs become extremely slow and the database grows excessively (around 850 MB per each million files). We perform full backups weekly and retain them for a year, so using this method would leave us with a repository database several terabytes in size by year’s end.

We are considering copying the different chains directly from the repository’s filesystem using “file to tape” (copying the vblobs, metadata files, etc.). This would greatly reduce the number of files being written to tape. However, it introduces coordination issues (we cannot run tasks on a chain while it is being written to tape) and it makes restores more complicated (we have to restore and recatalog the backup to access its contents—although for us, tapes are a last resort option in case of a total disaster).

Has anyone tried “backup to tape” using an entire repository as the source? Do you know if it behaves the same way (i.e., does it store the full file list)?
Has anyone else faced similar issues—needing to archive NAS volumes with millions of files to tape?

Thanks

Rafael

[david.domask]

Hi Rafael,

With regards to the database size, the system requirements are the same for both production direct to tape and backup of unstructured data backups to tape.

Long term we want to make this cleaner, but for the time being you will need to size as per the User Guide link I posted above.

If NDMP was alright for you, consider if there's a way to get the file shares with another image / volume level backup, for example VM or Veeam Agents for Windows / Linux. Those will not have the same database requirements that File to Tape has, but you will need to stage the backup to a repository first to do individual file restores.