Large-ish design with Scale-out repo and StoreOnce?

[david.buchanan]

Hi All,

I wanted to post my design here to get some feedback on it. Mostly to make sure I haven't overlooked anything but also to get some feedback on what others may have done differently.

We have two data-centres, DC1 and DC2. We have a very low latency fast link between them (10GB, less them 1 ms latency).

We currently have around 500-600 VM's which we need to backup on various schedules. Some backup once a day, others once an hour.

We have a 3PAR in DC1 connected to a HP Blade enclosure via FC. We are running VMware 6.0u2 on our Cluster. Once blade within the Enclosure is bare-mental windows which will be used as the backup proxy to use 3PAR storage snapshots.

These backups are written directly across the 10GB link to DC2, so no backup data actually lives in DC1 at any point.

My current design is to using existing storage we have and combine them into a Scale-out repo. This will come to around 54TB (24TB + 24TB + 6TB). This will allow us to fit all the VMs into the scale-out repo without any sort of de-duplication. These disk are also 10K SAS disks, so any restores or vNFS stuff we need to do should be nice and quick.

For our backup copy jobs (Weekly, Monthly, Yearly, etc) our clients require data be kept for 7 years. My plan here is to deploy a HP StoreOnce de-duplication device. I'm currently thinking I'll make use of the HP Catalyst support Veeam offers to improve GFS creation times. Now one issue I've run into here is that we currently use Backup Exec to write data to tape. However I've been unable to find a way to get Backup Exec to pull data from the Catalyst store to tape, I suspect this is not something Backup Exec can do? I do know HP offer a OST plugin but this is more for using a StoreOnce within Backup Exec to write backup data to the StoreOnce, not pull it out.

So, my other option is to use normal CIFS shares for the Copy job repo on the StoreOnce that way Backup Exec will be able to pull the data off the StoreOnce to tape. I know Veeam supports tapes and we do intent to move to it but for other reasons this won't happen before we deploy Veeam.

Any feedback or suggestions are welcome, even if it's design changes. I'm open to other products as well if better suited (eg: replacing the StoreOnce).

Thanks,
David.

[DaveWatkins]

You don't really want to try and pull from a Storeonce to go to tape, re-hydration will hurt performance and I suspect that's why there is nothing to do it. If you want to put data to tape, write it to tape from the disk repositories using either Veeam, or just writing the files to tape in BE.

Are you trying to use the StoreOnce as an intermediate stage and as stuff expires on it you then move it to tape, or are you planning to keep 7 years worth on the StoreOnce?

A word of warning on scale out repo's. Veeam isn't terribly smart about initial placement and has been known to just keep trying to put a new backup on a full extent rather than trying to put it on an empty one. It's also very easy to go from a normal repo setup to a scale out, but reverting back to a normal is significantly more challenging so maybe wait for 9.5 for scale out repos. I'm assuming the issues will be fixed then.

I'm assuming you've got Ent+ licensing to open up the storage snapshot functionality too

[david.buchanan]

Thanks Dave,

The problem with taking the data from the Scale-out repo is that it will only have the Backup files, it won't contain any of the Backup Copy data (weekly, monthly or yearly data).

The aim here is as follow:

1 - Scale-out repo - Has Backup Job data.
2 - StoreOnce - Has Backup Copy Job data (weekly, monthly, yearly data) - Because Veeam makes the weekly, monthly and yearly files as full backups the majority of these files will be the same (or very similar), hence the need for a de-dup set-up.
3 - Tape - Has backup of the Backup Copy Job data (weekly, monthly, yearly data).

In the event of something like crypto-locker or an employee going crazy and deleting everything then the tapes will still have the data off-site.

I understand the data will need to be re-hydrated to go onto tape, but I don't see a way around this.

The tapes and StoreOnce would contain all 7 years of data. It would not be removed from the StoreOnce once it's put onto tape.

We have Veeam Ent+ and the 3PAR snapshot licenses, so no issues here.

Thanks for the word of warning on the Scale-out repo data placement, I'll keep it on the list of things to watch out for. Does configuring the Scale-out repo to use one for full backups and the other for incrementals help with this? I assume it would?

[DaveWatkins]

The bit about the tape side is, what stops you from writing the tape when you run the copy job to the storeonce from the scale out repo? That'll get you the same data on tape as is on the storeonce without having to rehydrate, other than the extra time it ties up the scale out I can't see why you'd want to try and pull from the storeonce. Especially as you're talking about doing this with BE so your storeonce will have to be configured as CIFS and you'll lose the speed of catalyst.

Not really as that setting is per job or VM, so fulls are spread across the disks, and as mentioned, it's decision making is... poor. I tried it a few times on sites that would have really benefited from it. The sites it was designed for that had multiple repo's that were all say, 90% full and it made a complete mess of them, things like a scheduled active full that would easily fit on one extent was attempted against an extent is obviously would never fit on, and then that job would never succeed

[Gostev]

There were some bugs in the extent selection logic which were fixed in Update 1.

[david.buchanan]

Thanks Gostev, we are running update 1. So that's good to know.

Dave, Interesting idea, that might solve part of our problem. I'll have to think about that one a bit.

I guess my aim for the Tape is that is we lost DC2 then we could simply run a Active Full for all jobs from DC1 to get backups working. We would then restore all the weekly, monthly and yearly files from tape. If I do write the data from the Scale-out to tape then we don't have the actual W, M, Y files. Altho technically that data is all there as we would have the Backup job data from that point in time.

[DaveWatkins]

Ahh, I'm with you, you wanted to do GFS copy jobs and then backup those, that makes sense now, sorry I missed that. You could probably work something out manually to accomplish the same thing doing the backups at the right time directly from the scale out, but yeah, it won't be as clean. If you could use Veeam to write the tapes from the start that would make pulling them from the storeonce easier at least

[david.buchanan]

Yea, that's what I thought. Looks like going with CIFS on the StoreOnce is our only option at the moment. Once we move to Veeam for tape we can use Catalyst.

[DaveWatkins]

Just FYI, there isn't a migration option to move data from a Storeonce CIFS share to a Catalyst one I don't think

[david.buchanan]

Yep, I'm aware, but thanks. We will just have to start new copy jobs when we move. It would be nice if HP had some kind of Catalyst explorer to allow people to move files in and out of the stores....

[ChrisSnell]

David, take a look at the ExaGrid appliances. Half of the disk is used to land the backups and store them for about a week - giving a window of opportunity to push to tape without having to go through any kind of rehydration. This also assists with copy jobs as you're reading non-dedupe data.

There is also integrated Veeam code in the appliances, and of course there is also full support for Backup Exec.

https://go.veeam.com/exagrid-and-veeam.html

[david.buchanan]

Thanks Chris, very interesting product. I will take a good look at it.