Changing Backup Strategies with Backups and Backup Copies

[cfvonner]

I'm in the process of changing my backup strategies a bit. The way we have been doing it is thus:

1. Daily reverse incremental backups written to NAS storage (Drobo B1200i connected to Veeam server as an iSCSI disk). Currently I have 120-day retention of the reverse incrementals.
2. Daily backup copy to portable NAS storage (Western Digital MyBook Cloud 6TB). There are 4 of these drives in weekly rotation to an off-site storage vendor.

This has been working fine, but the recovery time from the Drobo B1200i is less than stellar. My primary VMware environment storage is a Nimble CS240 hybrid SAN device, and I have a fair amount of excess capacity (at the moment). So I thought I would take advantage of the performance and do the initial backups to a separate dedicated volume on this device. Here's what my new strategy looks like:

1. Daily reverse incremental backups written to dedicated Nimble SAN volume, with 14-day retention of the reverse incrementals.
2. Daily backup copy from Nimble to NAS storage (Drobi B1200i), with 120-day retention (and maybe adding some archiving rollups in there as well).
3. Daily backup copy from Nimble to portable NAS storage, drives rotated weekly, for off-site storage.

I've already disabled the current backup jobs (item a above) and created new backup jobs in their place (item 1 above). I then changed the source for the existing portable NAS storage backup copy job (item b above) to draw from the Nimble volume (so it is now basically item 3 above).

So, two questions:

1. Does this new scheme seem sound, or are there improvements that could be recommended?
2. Since I currently have 120 days of the existing reverse incremental backups stored on the Drobo B1200i, how do I automatically expire the reverse incrementals as they age past 120-days old?

Thank you,
-Carl V.

[veremin]

As to me, there is one caveat with the described scenario. Storing backup data to production storage makes you vulnerable to single point of failure. Should anything happen with the given device, you would loose both production VMs and their backups. At the end of the road you would still resort to drobo appliance and get stuck over restoration performance rates it provides.

Thanks.

[cfvonner]

As to me, there is one caveat with the described scenario. Storing backup data to production storage makes you vulnerable to single point of failure. Should anything happen with the given device, you would loose both production VMs and their backups. At the end of the road you would still resort to drobo appliance and get stuck over restoration performance rates it provides.

That single point of failure is understood. The overall goal here is twofold: improve performance of the initial backup job (writing to Nimble repository is much faster than writing to Drobo repository) and improve the performance of a recovery job of a single VM (again, the Nimble would be faster). As I said in my OP, step two of the new strategy would be to run a daily Backup Copy job from Nimble to Drobo; thus, in the event of a total storage failure, I would be falling back to the Backup Copy repository on the Drobo.

Also, within a few months, I will be setting up a DR site at our secondary campus and will have a second Nimble at that location which will be doing block-level replication of the VM volumes from the first Nimble (over a 1GB leased fiber-optic line). So I would also have the option of replicating back from the DR site once the primary Nimble unit was repaired or replaced. I'm not sure if I'll bother replicating the backup volume to DR.

Thanks,
-Carl V.

[veremin]

If the described scenario meets your requirements, and you don't bother that much about said problem, then, I don't see reasons why you can't implement it.

Thanks.

[cfvonner]

v.Eremin,

Thank you. I do appreciate the warning.

This new strategy actually came to me as a result of the Veeam presentation at Silicon Valley VMUG User Conference this past Wednesday.

[dellock6]

Carl,
I'll be really careful anyway in creating a first tier backup into the same storage, especially an hybrid array. By the way the I/O is managed in an hybrid array, you are going to face slowness and latencies decreases, and this is going to be worst since you are pushing the same backups into the array, thus creating additional I/O.

The problem is not only the "all eggs in one basket", but also the IO created onto the array. Especially when you need to run a full backup, an hybrid array is facing expected latency and slowness that you do not see at all when running VMs. This is because usually a VM has a "working set" that can be 10-20% of its size, and is made of the blocks usually used by the VM itself. Those blocks are saved onto the SSD part of the hybrid array, and served really quickly.
When you run a full backup however, we require all the blocks, even those coming from simple HDD. So my two reccomendations would be:
- never/never/never/ run the backup inside the same array, the controller will experience bad performances since it has to manage both reads from productions and writes to backup. With Reversed, this means a total of 4 I/O per saved block
- always use reversed incremental or forward with synthetic full, so the latency problems with active fulls only comes on day1 and you have then a "forever incremental". With reversed, run an active full every 2-3 months to avoid fragmentation problems.

Luca.

[foggy]

[*]Since I currently have 120 days of the existing reverse incremental backups stored on the Drobo B1200i, how do I automatically expire the reverse incrementals as they age past 120-days old?[/list]

And regarding your second question in the original post, if you do not need the entire backup chain anymore, you can right-click the corresponding backup under the 'Backups' node in Veeam B&R console and select the 'Remove from disk' command. This will remove backup files from the storage as well as cleanup the records from the Veeam B&R database.

[cfvonner]

And regarding your second question in the original post, if you do not need the entire backup chain anymore, you can right-click the corresponding backup under the 'Backups' node in Veeam B&R console and select the 'Remove from disk' command. This will remove backup files from the storage as well as cleanup the records from the Veeam B&R database.

I did know about the 'Remove from disk' command. That will remove the entire backup chain from the disk, not just the oldest ones. In looking at this more, it looks like I'll periodically have to manually remove the oldest reverse incremental backups from this old backup chain (to free up storage space) until I have three full months of the new backup chain on disk.

[cfvonner]

Carl,
I'll be really careful anyway in creating a first tier backup into the same storage, especially an hybrid array. By the way the I/O is managed in an hybrid array, you are going to face slowness and latencies decreases, and this is going to be worst since you are pushing the same backups into the array, thus creating additional I/O.

The problem is not only the "all eggs in one basket", but also the IO created onto the array. Especially when you need to run a full backup, an hybrid array is facing expected latency and slowness that you do not see at all when running VMs. This is because usually a VM has a "working set" that can be 10-20% of its size, and is made of the blocks usually used by the VM itself. Those blocks are saved onto the SSD part of the hybrid array, and served really quickly.
When you run a full backup however, we require all the blocks, even those coming from simple HDD. So my two reccomendations would be:
- never/never/never/ run the backup inside the same array, the controller will experience bad performances since it has to manage both reads from productions and writes to backup. With Reversed, this means a total of 4 I/O per saved block
- always use reversed incremental or forward with synthetic full, so the latency problems with active fulls only comes on day1 and you have then a "forever incremental". With reversed, run an active full every 2-3 months to avoid fragmentation problems.

Luca.

Thanks for the insights on the potential dangers. That has given me more to think about.

I'm less worried about the latency impacts (the backups all run during off hours when increased latency won't be noticed by anyone). So far, the Nimble array seems to be capable of handling the simultaneous read/write I/O without issue (my Veeam One monitor has not thrown any warnings about latency either, which it normally does whenever things get off kilter). This is probably because Nimble uses multi-core Intel processors on their controllers rather than more traditional controller chipsets found on most SAN devices, and can handle an enormous amount of I/O (I've pushed the array over 15K IOPS during testing without a hiccup).

I am using reverse incremental backups, so the backup windows (after the initial full) are also fairly short (and much shorter than they were when I was writing initial backups directly to the Drobo B1200i).

-Carl V.

[hmusa]

I can see Carl's usage as reasonable, from a certain angle based on limited hardware choice (expense... using what you have), and accepting that disaster recovery would resort to the slower storage in either case.

We have a similar setup. I have done a grand total of one full disaster recovery run (to test), but have accessed dozens smaller recoveries for various reasons (need a version of a file, need to rollback a bad software update on a VM). In these cases, having fast drive restore has great value.

Just to reiterate, I understand that ideally we have a spare machine that has the drives and can run independently from production environment. But in the real world we can't make that money appear. I will add that our DAS is subdivided into multiple arrays, and our production environment is running on 18 small fast drives, and I have a separate array of a few larger drives for the Veeam repository drive - so we're largely avoiding the reading/writing to itself issue.

[foggy]

I can see Carl's usage as reasonable, from a certain angle based on limited hardware choice (expense... using what you have), and accepting that disaster recovery would resort to the slower storage in either case.

Sure, it can be more or less reasonable in some particular real world scenarios, but from our side it wouldn't hurt to warn users of the possible consequences.