Comprehensive data protection for all workloads
Post Reply
CofIAlan
Influencer
Posts: 15
Liked: never
Joined: May 06, 2013 2:23 pm
Full Name: Alan Price
Contact:

Making copies of backups...without Backup Copy?

Post by CofIAlan »

I've been a lot of reading and research to figure out how to get us closer to 3-2-1 backups without going broke and while meeting some longer-term goals we have. It seems like everything I've read contradicts other things I've read. Hopefully someone can simplify things for me. I don't believe it's really as difficult as it seems.

Here's what we do today:
The most important part is that we have a dedupe appliance, a DD620 to be specific. We have plans to keep a dedupe appliance, though it may not always be a Data Domain… We take weekly fulls with nightly incrementals. This ensures that we have regular full backups without the overhead of synthetic transformations, which (this is important later) I’ve read really don’t like to behave with dedupe targets because of their random, 1-read+1-write IO needs. Right now our backups are taken from production and put on the DD where our retention policy keeps them around for a couple months. They then get magically cleaned up, one week at a time, by Veeam doing its thing.

Here’s what I want to do tomorrow:
Based on the Ultimate Backup Architecture blog post and follow-up webcast, I want to introduce a new primary storage target that’s fast, small, and cheap. From there the backups will move to a dedupe appliance for retention. Primary backups would be kept a week or two while deduped backups would be retained a few months or more. We’d be able to maintain RPO, improve RTO, lower TCO, and shrink our backup window. All the important buzzwords are achieved.

Here’s the problem:
I can’t figure out how I’m supposed to do this. I know that it can be done, but how do I do it efficiently? Backups should be captured during their window and the copies should be made during their window. The first solution is a group of Backup Copy Jobs. The second is a custom PowerShell script.

Pros of using Backup Copy
  • Veeam handles the movement of data from primary to secondary storage
    Veeam handles the two different retention policies for each “type” of media
    Veeam moves only new or changed blocks from primary to secondary regardless of whether the source file is full or incremental, which means data should move faster
    Veeam is my backup platform and all my backup tasks stay inside Veeam
Cons of using Backup Copy
  • Veeam moves only new or changed blocks from primary to secondary regardless of whether the source file is full or incremental, which means transforms have to be performed to get synthetic fulls
    Veeam documents and forums recommend against using and synthetic operations with dedupe appliances because of the heavy, random IO workload
    Veeam Backup Copy Jobs are literally always running and like to log failures when the server reboots (this is more of an annoyance…I personally would prefer if the jobs, which we’d only allow to run when we’re not doing production backups anyway, had a true scheduled start time and ran like a normal backup job)
Pros of PowerShell
  • I can do exactly what I want without any changes to Veeam and on my backup window schedule
Cons of PowerShell
  • I have to write and maintain the whole script
    I might be duplicating features (that I can’t figure out) and wasting time and resources
    I must move every single new backup file, which could be slower than intelligent Backup Copy Jobs
In a perfect world I could accomplish what I want with Veeam: strict copies of backup job files that dump the data I have from fast, quiet (non-production) storage to a long-term dedupe appliance or archive, combined with scheduling and retention periods for the copied job files. This is basically replicating backup files instead of VMs. I like this balance because I can move production data to primary backup as fast as possible, then move primary to secondary on my time, all without the overhead of special transformations.

(Note: I did review a forum post at http://forums.veeam.com/veeam-backup-re ... t1860.html from a couple of years ago where someone was asking about simultaneous targets. The point Gostev made was that it might be doable [in that particular situation] but the overall backup job would only work at the slower of the two targets. So while read-once-write-twice would be neat it still leaves a potential bottleneck. But, maybe this is still an option if enough customers have two targets that can both keep up with the single read from production storage. We currently fit our backups into our backup window, so we’d be able to take advantage of it even if it’s not the FASTEST option for us. Of course, then I’m looking to the future where we might be outside our backup window and need the speed WITH a dedupe appliance still in the picture…it’s a vicious cycle when you have to save money. Otherwise I’d just own a bunch of pods like https://www.backblaze.com/blog/cloud-storage-hardware/.)

So then my distilled questions:
Can I make Backup Copy Jobs play nice with dedupe storage? Am I overthinking it and everything will actually be fine (besides those quirks of always-running jobs that bug me)? Or do I need my own script that lets me do what I want but not within Veeam itself?

Any and all thoughts are much appreciated.

Thanks!
Alan
Shestakov
Veteran
Posts: 7328
Liked: 781 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Making copies of backups...without Backup Copy?

Post by Shestakov »

Hello Alan,
I believe backup copy job is what you need since it`s made exactly for goals you are are aiming.
CofIAlan wrote:Veeam moves only new or changed blocks from primary to secondary regardless of whether the source file is full or incremental, which means transforms have to be performed to get synthetic fulls
Veeam documents and forums recommend against using and synthetic operations with dedupe appliances because of the heavy, random IO workload
Backup copy job doesn`t do synthetic fulls, it works alike forever forward incremental method.
CofIAlan wrote:Veeam Backup Copy Jobs are literally always running and like to log failures when the server reboots (this is more of an annoyance…I personally would prefer if the jobs, which we’d only allow to run when we’re not doing production backups anyway, had a true scheduled start time and ran like a normal backup job)
Does it log a lot? By the way, if you don`t want it to copy files during some periods of time, you can enable backup window.

MORE Pros of using Backup Copy
  • -GFS retention as well as basic one
    -WAN aceleration
    -Healthcheck
    -Notifications
    ... much more
So again, there are much more pros of backupcopy.

Thanks!
CofIAlan
Influencer
Posts: 15
Liked: never
Joined: May 06, 2013 2:23 pm
Full Name: Alan Price
Contact:

Re: Making copies of backups...without Backup Copy?

Post by CofIAlan »

Thanks for the reply Shestakov.
Shestakov wrote:Backup copy job doesn`t do synthetic fulls, it works alike forever forward incremental method.
Ok, I used a bad term there. We don't use synthetic fulls in Veeam-speak since we have active fulls.

However, I believe my main concern is still valid about Backup Copy Jobs: they must be transformed (see https://www.veeam.com/kb1932) to produce their anchoring full backup. This transformation process is very random IO intensive. According to many of the docs and posts I read this IO pattern has very poor performance on dedupe appliances (specifically those that do full inline dedupe [like DD] without a hot landing zone [vs. ExaGrid]). My understanding is that it's a mix of how quickly the data can be rehydrated and what the data is doing. Sequential reads can be optimized because of read-ahead functions but random IO has to jump all over the array to find the right blocks.

In fact, a Veeam KB, https://www.veeam.com/kb1745, specifically discusses this idea:
Forever Forward Incremental, Synthetic Full, and Reversed Incremental operations will perform poorly with deduplication devices because the backup files that have been written to the storage previously must be interacted with during these operations. The act of reading existing data on deduplication device is slow because each block requested must be rehydrated and uncompressed to be read.
If transformations with deduplication are a bad idea is Backup Copy still the right option?
Shestakov
Veteran
Posts: 7328
Liked: 781 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Making copies of backups...without Backup Copy?

Post by Shestakov »

You are correct Alan, but it`s still looks more wise to use backup copy job over the script. Transforms are slower on dedupe performance, but I`ll give it a try to see how it works in your case.
You can consider File copy job as an alternative of the script. if backup copy job shows bad performance in your infrastructure.
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Making copies of backups...without Backup Copy?

Post by veremin » 1 person likes this post

Alternatively, you can wait till v9 is released and use backup copy job with periodic "active" fulls, avoiding any transformation activity on dedupe appliance. Thanks.
Delo123
Veteran
Posts: 361
Liked: 109 times
Joined: Dec 28, 2012 5:20 pm
Full Name: Guido Meijers
Contact:

Re: Making copies of backups...without Backup Copy?

Post by Delo123 »

"No transformation" is the magic word for all our real world problems regarding dedupe-appliances. Go V9! Can't wait...
CofIAlan
Influencer
Posts: 15
Liked: never
Joined: May 06, 2013 2:23 pm
Full Name: Alan Price
Contact:

Re: Making copies of backups...without Backup Copy?

Post by CofIAlan »

Hi all.
I disappeared for a bit between projects. I remembered to go looking in Veeam tonight and found the new option in V9 by accident, forgetting this post. A quick internet search returned the latest and greatest user guide pages. I'm currently testing it out in a long-term simulation I have running (preparing for next year's budget) but so far so good. Very nice to have this feature and I'm looking forward to seeing how it does!

http://helpcenter.veeam.com/backup/vsph ... _full.html
http://helpcenter.veeam.com/backup/hype ... _full.html
Shestakov
Veteran
Posts: 7328
Liked: 781 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Making copies of backups...without Backup Copy?

Post by Shestakov »

Hi Alan,
That`s correct, backup copy active full is part of v9 release as well as 100+ other features.
Post Reply

Who is online

Users browsing this forum: mberrey and 156 guests