Best way to backup a NAS (with a huge amount of small files)

[paruccone]

Hi!

I am running a lab at home. In the past I bought a cStorageloader LTO-4 library. Sometimes it was working fine some times it wasn't, probably HW issues with the host.

I created my backups using an LSI SCSI controller that allowed a block size of 1MB out of the box. As I had issues with restoring files, I thought something might be wrong with the HW. Now I have what seems to be a very stable HW environment: Proliant DL380e Gen8 with HP LSI SCSI controller to the storageloader and HP FC4 SCSI card (Emulex) connected to an MSL2024 LTO-4 library.

Problem number 1: FC/MSL combo was proposing max 512K block size preventing me to read tapes written be LSI+StorageLoader. It took me weeks to understand how to fix this (there some docs for QLogic HBAs but extremely difficult to find anything for Emulex). Here's the answer (there's also a command-line tool called HBACLI for both Windows and Linux allowing same thing): OCM (One Command Manager)->adapter, select card->select port, go on Driver Parameters, change ExtTransferSize : default is 0 = 512KB, 1= 1MB, 2=2MB, 3=4MB

Problem is: tape catalog takes more than 20h for each tape. I suppose this is due on how VBR writes and reads the catalog from tape as it happens no matter which hardware I use (and with no errors). The backup have been done using file backup with a Linux Agent on the NAS VM.

Is there a more efficient way of doing the backups that will allow for faster read of the catalogs in case it is required? Backing up the VM?

My backup spans 4 to 5 tapes (about 3-4TB) which means that only to read catalogs it takes about 80 hours!

PS: the issue appears on tapes containig lots of small files, not while cataloging first tape containing few huge files.

Thanks!
Alessandro

[PetrM]

Hi Alessandro,

Looks like that the backup of VM might be one more option.

But I recommend to consider NAS backup as a method to backup a huge amount of small files, might be exactly what you need.
You can find precise instructions for implementation on our help center.

Thanks!

[paruccone]

Thanks for you quick answer Petr! Looks like this is a new feature implemented with v10, right? Asking so as I started using VB&R back at v9.0.

I'll give it a try in the coming days but, despite many great features for enterprise users like versioning, etc. (which are not applicable to my case), I don't get the implications of using it in order to backup to (and restore from) tape: looking at the link you sent I see no mention of it except, maybe, for backup repository and archive repository & long/term retention.

Any detail&white/paper/docs available? My feeling is that this introduce some more requirements (proxy, cache, short term retention) but I don't get the advantage in restoring from tape.

Thanks!
Alessandro

[PetrM]

Hi Alessandro,

Yes, you're right, this is one of the new features in the version 10, it might be useful if you decide to review your backup strategy in future.

If you need to backup files directly to tape: file to tape job is an option but catalog can take significant time.
If you have a primary storage for short term retention: you can run VM backups and schedule backup to tape jobs.

Thanks!

[paruccone]

Hi Petr!

Now that the catalog is done and I am restoring, I noticed another issue: usage of SQL memory/system/DB disk IOPS.

The restore has been now running for 15 hours and 2.4TB out of 3.7TB are restored.

Doing a full restore and veeam is hitting extremely hard on SQL (not express): as this was a basic installation with no tuning, and no other DB installed than Veeam on it, the tape and (restore) disks are sitting most of the time waiting for SQL which is generating a huge amount of IOPS on the system disk. So sequence being: SQL waiting for I don't know what generated by veeam, read tape->write disk->tape stops reading, back to step 1. SQL hitting max 2GB memory, tried some tuning options but I suppose I will have to restart the service (Windows server hosting SQL+Veeam has 16GB of RAM of which 6.8GB only are in use, 2GB from SQL process).

And this on a configuration with 1 server and 2 agents; I can understand that VB&R may not be designed for this use scenario but:
- you may want to put a warning when backing-up/restoring a certain amount of files (I've seen in the logs that huge strings containing all the file names appear in the log) saying that this may impact system performance or pointing to a tuning guide< but I suspect it might be related to this: https://social.msdn.microsoft.com/Forum ... progeneral
- document how to parameter veeam/SQL (Lock page on memory, min/max ram?) to work better under those circumstances

I don't see a short-term retention as a viable option: that would simply double the storage space required by the NAS if I understand that correctly. Doing better use of SQL memory considering it-s not passing the 2GB threshold and that there are more than 9GB available would be a better option IMHO.

I'm sure I am missing some tuning (perhaps the min memory will take effect only after restarting SQL service?).

Thx,
Alessandro

[PetrM]

Hi Alessandro,

Thanks for your suggestions and detailed feedback!

1. I assume that SQL load is growing proportionally to the number of files because of catalog operation which is performed under the hood via SQL queries to the Veeam database, look at the description of restore logic. I would take a look at most expensive queries but if you would like to perform a detailed technical investigation, I'd recommend to open a support case and ask our support engineers to help you to determine the root cause of this behavior. From my point of view this behavior is expected but it is always better to double check.

2. There is the best practices guide which contains some tips for Veeam database sizing and information about possible I/O increase when file to tape jobs are extensively used.
I think it's a good idea to add some specific values to this page but we need to collect some statistical data, it's not a quick task.

3. Short term retention consumes less space because of compression and deduplication operations performed during backup.

4. SQL caches data memory for performance purposes and the default behavior of the database engine is to acquire as much memory as it needs. I think all main options for memory management are described in this article (most likely you've already seen it). In fact, I don't think that you're missing some serious tuning (service restart is worth checking).

Thanks!