Veeam Backup server+ Storeonce+ MSL6480 backup

[Hirosh]

Hi Guys,

i know this might seem simple, but i have to ask this:

we are working on a backup solution which comprise of a physical Veeam Backup server, a storeonce for disk based-backup and a msl 6480 lto9 tape library for tape backup. what is not clear for me is that, shall i add the tape library as a VTL in StoreOnce or i should follow the common way of adding the tape library as a repository in Veeam Backup server. if i can do either, what are deciding factors to decide which one to use?

best regards,
Hirosh.

[Regnor]

You can't connect the MSL to the StoreOnce. While the StoreOnce itself can host a VTL, it can't target a physical tape library. You will have to connect the MSL to your physical backup server and setup backup to tape jobs.

Something else. Does your physical backup server also have local disk space? Or do you plan to only use the StoreOnce as a repository?

[karsten123]

I wouldn‘t recommend a StoreOnce as primary landing zone for your backups.

[Hirosh]

You can't connect the MSL to the StoreOnce. While the StoreOnce itself can host a VTL, it can't target a physical tape library. You will have to connect the MSL to your physical backup server and setup backup to tape jobs.

Something else. Does your physical backup server also have local disk space? Or do you plan to only use the StoreOnce as a repository?

Hi regnor,

one more thing to confirm, is it possible to create a backup copy job with the source being the backup to the storeonce job and destination being the Tape Library? or i have to go through the backup to tape wizard and choose the backup to Storeonce as the source? or when i create the main backup job choose " Configure secondary destination for this job" and choose the tape repository as the secondary?

[Hirosh]

I wouldn‘t recommend a StoreOnce as primary landing zone for your backups.

as the matter of the fact our primary landing zone would be an HPE storage , then we would use backup from storage snapshot to backup to StoreOnce.

[Regnor]

You would have to setup a backup to tape job. Backup copy only works for disked based storages. But you wouldn't want to use the StoreOnce as the source for your tape backups. Deduplication appliances have a really low (random) read performance and I'm almost sure that you won't be able to write anywhere full speed to tape.
Why don't you use the HPE storage as the source? Or is this your production storage?

[Hirosh]

The HPE storage is Production Storage so we can not add any more load on it, we add Storeonce to :

1. add a disk-based backup as re-assurance
2. a faster backup compared to Tape
3.to use it as a source for tape backup at another schedule(time interval) so we don't involve our production storage
4. use its catalyst copy feature to a replica site(another storeonce).

we gonna use Storeonce 5660 with 8 SSDs as cache, would that still give us poor performance?

[Regnor]

Ok, that wasn't clear to me as you referred to your HPE storage as the primary landing zone.
It's not recommended to use a deduplication appliance as the primary backup repository (like Karsten advised).
You may have a good backup performance, but restore performance will be degraded; same goes for backup to tape performance.
In general you should have at least a fast primary repository for short-term restores, and use a deduplication appliance only as a long-term storage.
I can't comment on the StoreOnce 5660, but I just wouldn't go that way.

What retention are you planing for your backups? And why do you want to go for a StoreOnce instead of a low-cost storage server (you already have the VBR server physical)?

[Hirosh]

uhmm, i think you have misunderstood my scenario. The data is stored on a HPE Primera , it would be backed up continuously to storeonce which will remain there for longer period. Meanwhile there will also be a backup to tape job to create another copy on Tape library. also data will be replicated to another Storeonce which is located in another site with catalyst copy. We are planning to follow 3 2 1 rule. We will be keeping the data for long periods. The Storeonce Appliances will be expanded with multiple chassis. we would like to utilize the deduplication & Catalyst copy features of Storeonce.
are we suppose to use another Fast storage in-between HPE Primera and Storeonce? why would the back up to tape be slow?

[Regnor]

That's at least what's recommended. Normally you have a fast first tier from which you can restore in shorter time. All long term backups go to the deduplication appliance, where restore performance isn't that critical anymore.
Over time the restore performance can decrease. You might not be able to achieve your planed RTO or create enough IOs for the tape offloads.

Perhaps someone from Veeam can tag 'FedericoV', as he'll be able able to give some more insights on the StoreOnce systems.

[Hirosh]

yes I would much appreciate if I could have more insights on this, how can i have 'FedericoV' or anyone from Veeam's comments on this?

[Hirosh]

guys , any more comments on this?

[Thommo]

We recently purchased an MSL6480 as well, it connects directly to a physical host via FC and we use a Netapp FAS2720A with 500TB of spinning disk. It works very well with Veeam, the four tape drives allow you to run multiple jobs at the same time

[Hirosh]

Hi Tommo,
interesting point i have also experience working with netapp. on your FAS2720 im assuming you are using 4*SSD drives as cache for your aggregates(if even not using flash cache on the controllers) & Netapp controllers have impressive throughput considering they use WAFL. Here we are concerned the Storeonce be the bottleneck not the Tape library.

regards,
Hirosh

[HannesK]

Hello,

I could have more insights on this, how can i have 'FedericoV' or anyone from Veeam's comments on this?

I can only repeat what the others already said, and Federico would say the same: go for Apollo (or any other dumb block storage) as primary backup target and from there you can do backup copy jobs to StoreOnce and / or backup to tape jobs to tape

A recommended setup looks like this
Primera as production storage ---- backup job -----> Apollo ---- backup to tape job ----> MSL
Primera as production storage ---- backup job -----> Apollo ---- backup copy job ----> StoreOnce

FAS2720A with 500TB of spinning disk. It works very well with Veeam,

good to hear. FAS systems are relatively slow and expensive compared to E-Series or any other "dumb" block storage. So good to hear, that there are customers happy with that solution

Best regards,
Hannes

[Hirosh]

Hi Hannes,

thanks for the recommendations. just a friendly reminder FAS Series are not "dumb" block storage, They are unified solutions and have quite impressive throughputs, E-series provides only Block storage and almost none of netapp Advance features(which is out of scope of here) thats why E-series are way more cheaper. but you are right as a "dumb" block storage it would be wise to go for E-series since the core of E-series is speed with extremely low latency controllers and data paths.

best regards,
Hirosh.

[HannesK]

Hello,

just a friendly reminder FAS Series are not "dumb" block storage

yes. That's what I said That's why I recommended E-Series. Veeam runs best on "dumb" block storage. Price / value / performance ratio is much better and the advanced features are not needed for a backup storage

Best regards,
Hannes

[FedericoV]

Recent StoreOnce SW releases are really more efficient in restore speed. The overhead of the deduplication layer is lower than someone can expect. The metadata lookup occurs mostly in RAM, and metadata itself is on the SSD volumes, so the added latency is small. When we run a restore, the real data come from spindles, and StoreOnce adopts 16TB disks, so you cannot expect SSD-like speed and latency, but do not underestimate the performance you can get. The cumulative restore speed from StoreOnce is close to the restore speed you could see from an equivalent HW running Linux and having a XFS on RAID60. This to say that the deduplication layer is now much lighter than it was in the past. With StoreOnce, a Full restore is faster than a file or a dir restore because it is a sequential operation and StoreOnce can make a good read-ahead to speedup the read process.
Now, we all know that Veeam is one of the best backup solutions (probably the best) for Storage Snapshot Integration, and VBR integration with HPE storage is one of the most complete in the market (I guess it is the most complete). For this reason, I often suggest to include the Veeam "Snapshot-Only-Backup-Jobs" in the our protection policies, along with the normal nightly backup to StoreOnce. They are light, fast, and can be executed every few hours. To avoid using too much primary storage capacity, it is better to keep the snapshot retention shorter than one week. Even a 3 days retention is usually enough for our goals. Based on my experience as backup-admin, most of the urgent restore requests are for fresh data, i.e. for data still in retention on a storage snapshots, and RPO is great too. Restoring from storage snapshot is extremely fast, and even IVMR is as fast as the production VM because the ESXi server mounts the snapshot as a normal storage volume and does not require vPowerNFS.
That said, once we adopt Storage Snapshots, restore operations from StoreOnce are less frequent, but they still run at a very reasonable speed, especially for "backup-copy to tape" jobs where the source is on StoreOnce.
Recently I did a performance tests with StoreOnce (a 5260, with its minimal disk configuration), VBR V12-beta, and an LTO-9. My goal was to verify that the process pipeline: StoreOnce --> Tape-Server-->LTO-9 was reasonably fast, and in particular, fast enough to keep the LTO-9 in streaming state without causing the bad "shoe-shining" effect. Remember that when VBR writes to a tape, it makes a single stream operation, reading from the backup-repository, and then writing data to the tape device, so here what we need is good single stream performance. Well, in my test the tape job run at 430MB/s to a single tape as reported on VBR GUI. This is not the maximum StoreOnce performance that you can achieve with multiple concurrent restore streams.
To understand whether this throughput was good or bad, I tested an equivalent task, but this time the source backup-repository was on the Tape-server itself, and on local NVMe disks, i.e. on the fastest configuration I could implement in my lab. In this case the tape job throughput was 570MB/s. Obviously it was faster than before, but the difference was not so big - and I think it doesn't justify backup to NVMe
I didn’t make any tuning to try to improve performance, I just wanted to make sure my small StoreOnce, even in a configuration with the minimal number of spindles, was fast enough to keep the LTO-9 in streaming state, and it worked!

P.S. Did you know that with StoreOnce software 4.3.2 and VBR V12, on single stream backup (i.e. a backup job with just one VM) can ran at ~1GB/s (i.e. >4x faster than on V11), and even the synthetic full is 2-3x faster than before? This is important because sometime we have to protect huge-monster VMs, and for them the cumulative throughput is not relevant, what they need is high single stream performance.

I hope it helps
Federico