Backup Copy Job to dedupe appliance

[ckbrou]

With the new backup copy job feature I am hoping to change my backup strategy and am looking for some direction. I have a QNAP NAS onsite and 2 EMC Data Domain DD620 dedupe appliances. One DD620 is onsite and the other is offsite connected via a 100 mbps link.

Here is what I am hoping to do:

1 week of daily reverse incrementals to the NAS
Backup copy job to the onsite DD620 saving the most recent daily and 4 weekly backups. Since the DD620 replicates offsite this gives me an offsite backup each day and also allows me to keep 4 weekly copies to meet our retention policy.

The other option would be to take both DD620 appliances offsite, use them both for even more archival storage utilizing the backup copy job to send the data offsite from the start. The DD620s can't be "joined" as far as I know so I am not sure how that would work.

Any thoughts? Since I don't fully understanding how the backup copy jobs work I am looking for some direction.

[Vitaliy S.]

Hello,

Backup copy job copies actual VMs from the backup files, so you can use multiple jobs, backup files as a source for your backup copy job. Your first scenario seems a bit more reasonable to me, as you will have 4 weeks of data onsite that might be handy when the disaster strikes. Also having another DD appliance offsite will protect you from losing the entire source site.

If I were you I would configure backup copy job in a direct mode and point it to the onsite DD appliance, WAN acceleration will not be required in this case.

Thank you!

[ckbrou]

I will give that a shot. It will be ineteresting to see how the DD620 handles a backup copy job vs. the forward incremental jobs I have been sending to it with my previous strategy.
Thanks for your advice!

[Tibo]

I tried Backup Copy Job with a StoreOnce D2D physical appliance and I have to say that it's not a good option.

Backup Copy uses a sort of Incremental mode where Full backup file (.vbk) is usually merged every day with the nearest increment file (.vib).

This means that the dedupe appliance has to rescan the whole VBK file in order to find what blocks are new. Since most dedupe appliances have a low read IO throughput => it takes forever.

Your StoreOnce will also try to sync the VBK file everyday and will have only 24 hours at best to do so. Every day the process starts from beginning (Big VBK file to rescan + a new VIB).

In the recommended mode (standard backup job with incremental + active full) you don't have this scenario. The VBK and VIB files won't change once backup is done. You D2D will only try to replicate the new VIB file (and the active full's VBK once a week/month, usually during the week-end).

@Veeam Team : Please add an option to select the backup copy mode, it would be more appropriate to use with dedupe appliances.

[Vitaliy S.]

Hi Tibo,

Current logic/mechanism was implemented to allow you to save space on the archival storage. Usually, the retention policy in these scenarios is a long one and with standard approach, forward incremental backup mode + active fulls, you will consume a lot more space on the destination storage. So it's the matter of trade-offs, assuming that backup copy jobs does not "touch" your production VMs, then the time required to build the synthetic full should not be that critical. As a workaround, you could potentially use multiple backup copy jobs, in this case the load on the dedupe device should be less (assuming backup copy jobs do not start transformation process at the same time).

Thanks for the feedback!

[Tibo]

I think you may not have fully understand my previous message. I actually don't care how much time Veeam needs to create a synthetic. My issue with backup copy job is that it creates a synthetic everyday which means that the full backup files are modified everyday too. This is a very bad approach for some (if not most) dedupe appliances. It would be great to be able to make a synthetic only once a week or month instead.

Why do backup jobs have an option to select backup mode (incremental, reverse, with synthetic or not ....) and not copy jobs ? That would make sense to add it in a future release for this kind of scenarios.

[ckbrou]

I tested the backup copy jobs with my data domain dd620 and ran into the same issue at Tibo. The copy job performance was very poor on dedupe storage. For me, the synthetic creation time did matter because it was taking more than 24 hours (when the next backup jobs needed to run) and the dd620 was overloaded at that point.

Although I really like the idea of the copy job, it still ended up being much better for me to run two separate backups. One to my QNAP NAS and one to my DD620.

[mongie]

Two things I've noticed...

1. Despite the fact I dont have Veeam Dedupe Enabled, there is a status on some VM's within my Backup Copy Jobs (when they from a separate backup job to the bulk of my VM's) that says "Restore point is located in backup file with different block size".

I don't really have a problem with it, and I realised what the issue was once I saw the "action" (One job set to Dedupe Disabled, Local Target and the other set to Dedupe Disabled, LAN Target) but this should be shown as an error, because my job has been running for 20+ days now, and I hadn't had any warning that one of the VM's wasn't being copied.


2. There needs to be some sort of logic that deletes backup copy files if there is never any backup data copied into them... I'm being left with a whole heap of 16MB files that are created on the days where I don't run the underlying backup jobs... I only backup M-F but there doesn't seem to be a way to tell the backup copy job about that... With 80+ daily restore points, and 8 backup copy jobs, it adds up.

[Gostev]

1. This has nothing to deal with dedupe, but rather due to using different storage optimizations for different primary jobs. Either set all your primary jobs to use the same block size (and do Active Full), or use a separate Backup Copy job for each primary backup job.

2. We will consider this as a future enhancement, although we are talking about saving probably 0.0001% of your backup repository space (estimation based on 16 MB file per 1.6 TB of backups). Even if 0.01%, which is 100 times more, still sounds like very little value to the product for all the work and testing involved.

Thank you!

[jbarrow.viracoribt]

Thank you for this thread. For the life of me I've been trying to figure out why we have such a high change rate on our TwinStrata Cloud Array appliance using backup copy jobs. I'm disabling that right now thanks to this thread.

[seo]

Agree, I have also tried a scenario with copy jobs to DD620 and I have now disabled and deleted jobs. Useless against my DD620, like all synthetic operations are.

[romura]

I also ended up disabling my "backup copy job" to a Windows 2012 deduplicated volume. The daily transform operation was taking way too long to rehydrate data and space became an issue as the data re-hydrated. I guess the only way to run a copy job to a secondary site with deduplicated storage is to have a second backup job? I wish there was an easier solution for this method as deduplication seems to be becoming the norm for backup storage. I like the idea of having a copy job run off backups jobs already completed, but it won't work in my scenario. Having two backup jobs for the same VM seems very inefficient.

[Vitaliy S.]

Hi romura, were you using GFS or regular retention policy for your backup copy jobs?

[romura]

We were using a regular retention policy set for 60 days. What we noticed is that after the 60 day retention mark, the .VBK file will transform daily which took a long time and eventually had to eat up all the space due to re-hydration. We have numerous .VBK files that's almost 1TB each in this repository, so imagine all these 1TB .VBK files that will have to transform daily on this dedup'd volume, basically breaking the dedup cycle.

If Veeam is touting Windows 2012 deduplication, at least put a disclaimer not to use it with Backup Copy jobs. I basically lost 60 days worth of backup copy jobs at my DR site since I had to restart the cycle with regular incremental backups with periodical active fulls.

Is there another way or a best practice to do backup job mirroring on dedup appliances that doesn't require two backup jobs on production data?

[Vitaliy S.]

Well...I believe it's not only about deduplication, but also about hard drives (their performance) that are installed on this Windows box. If you don't want to hit your production VMs with two backup jobs, then you may want to continue using "old-school" method of using Robocopy and other scripts to push data to a dedupe volume.

Also it seems like this Windows Server behavior (processing rate of 100 GBs per hour) could also be the reason for this slowdown.

[romura]

I understand the Windows 2012 deduplication and the underlying disk performance factors, but the point is the topic of this thread. There shouldn't be Backup Copy jobs to dedup repositories and Veeam should state that in some way. We're talking about TB's of data that has to re-hydrate every-single-day which is a flaw of Backup Copy jobs coupled with deduplication appliances/servers.

[Gostev]

We're talking about TB's of data that has to re-hydrate every-single-day which is a flaw of Backup Copy jobs coupled with deduplication appliances/servers.

I don't believe it is correct to generalize this Windows 2012 issue. Most appliances will actually only re-hydrate the blocks that are being accessed. Some dedupe appliance will even keep the most recent data in the landing zone, and only dedupe GFS fulls. I am surprised Windows needs to re-hydrate the entire file, this does not seem optimal to me!

[chrisdearden]

When using server 2012 dedupe , I have found that as long as you only try to dedupe GFS points by setting the "dedupe files older than" setting to older than the regular retention period will allow it to work on those GFS points, which do not change once created.

[Delo123]

@Veeam. I think i asked for this before, but since more and more people are using 2012 Dedupe it would be really nice to be able to let veeam "split" Backupfiles at a certain size.
Creating smaller Backupjobs is not always an option, and creating seperate jobs for disks within a VM is a management nightmare.
A per job setting of: Split Backup Files in "selectable value" Chunks would be great. A good default would be something like 500gb since this would still be "ok" for Windows Dedupe.
The way it is now, if windows cannot dedupe the 3,4,5 or maybe 10tb backupfiles it will start over again, which is simply stupd...

[Vitaliy S.]

Hi Guido, thanks for your feature request. While "splitting" sounds like a reasonable solution to overcome dedupe engine limitations of some appliances or Windows Server, for me it sounds a bit scary, as this will increase the number of backup files to maintain. There will be chances of losing a single file that will prevent you from using the entire restore point. Just my 2cents.

[Delo123]

Hi Vitaly,

I understand what you mean, butbnow we already have a lot of files to take care of (incrementals), and believe me i rather copy 100x 200gb files instead of 1x 20tb file... Also you can use multiple threads etc... Anyway right now we are splitting veeam backups with .arj archives which is much much worse... Anyway nodoby "has" to active it but please make it optional

Thx, Guido

[Vitaliy S.]

Yes, I agree, that sounds worse, as you have to manually control the retention policy and other things. Thanks for your feedback.

[luckyinfil]

good thread. Looks like the only workaround for having duplicate backup jobs with dedupe appliances is to have a separate backup job writing to the dedupe appliance to forgo the transformations.

[ckbrou]

I am probably stating the obvious here and this may not be possible for many but I found a solution for me. I ended up putting a Data Domain device onsite, pointing my regular backups job to it. Then I am using a NAS offsite as the backup copy job repository. Since all of the synthetic operations happen on the backup copy job repository this is working much better.

This allows me to keep 90 days of backups on the Data Domain locally and a few days of backups offsite on the NAS if ever needed.

[haslund]

2. We will consider this as a future enhancement, although we are talking about saving probably 0.0001% of your backup repository space (estimation based on 16 MB file per 1.6 TB of backups). Even if 0.01%, which is 100 times more, still sounds like very little value to the product for all the work and testing involved.

Hi Gostev,

I completely agree from a space saving perspective, 16 MB means nothing.
However, what I do not like is that is my backup job fails, having not even backed up a single VM! Then Backup Copy Job will still create an "empty" VIB file and start merging the oldest vib into VBK.
This means I am loosing old backup data without saving new data... not too ideal in my mind.
Maybe some logic to validate if new VIB actually contains any data, if no then skip merge/inject process?