Veeam 8 - Forever Incrementals and Deduplicating Appliances

[ChuckS42]

Recently upgraded to Veeam 8, and am rather enamored of the Forever Incremental idea. However, when the merge process hits it wreaks havoc with my storage - a Dell DR6000 appliance (performance goes into the toilet, other backup jobs crash out, etc.). I now realize this is because the merge process is a kind of synthetic full backup, and doing those on deduplicating storage can be bad. I don't see any references to this sort of thing in the Veeam 8 documentation or release notes. Is it just me, or is this behavior expected?

[veremin]

This method has been appropriated from the backup copy job – the oldest restore point start to transform, once the retention is reached.

If the said storage can’t cope with transformation activity, you might want to switch to time proved forward incremental mode with regular active full backups. Or land your backups on some fast local storage first and then copy them to dedup appliance to guarantee long term retention.

Thanks.

[ChuckS42]

Very interesting. I will take a look at that article. Thanks!

[nikpolini]

HI Chuck, just wondering what results you have been having with the DR6000 and Forever Incremental. I ask as my desired setup is as follows:
Daily forever Incremental with 14 day retention to DR6000, then a weekly backup to tape job using the Virtual Synthesized Full to tape to ensure only a single latest backup of a vm goes to tape for long term offsite.

In order to use the Virtual Synthesized Full I need to use Forever Incremental. How is your DR6000 going with these (VIB role into existing VBK) or did you find a different solution?
Cheers
NikPolini

[ChuckS42]

Do NOT use anything even remotely synthetic on the DR6000. I had to scrap the FFI and go with a traditional weekly Full/daily Incremental model. If you use FFI, when the incremental-merge happens the DR6000 must completely re-hydrate both the Full and merging Incremental, then immediately re-dedupe and compress the file as its written. This creates a huge performance problem for the array and the backup job will likely fail, especially if you have other backup jobs running as well.

It took a long time to clean up the mess running FFI backup jobs created. I've had a highly-escalated case with Dell open on this (and other) issues for a few months now, and we're finally mostly stable. They've been releasing code updates to help some of the problems I've encountered, and another is due soon. I've been told they've reached out to Veeam to get more details on how they read and write data, so maybe there will be some Veeam-related optimizations to the DR6000 code in the future that will allow the use of FFI, but I'm not holding my breath.

[nikpolini]

Thanks Chuck, that info is much appreciated. So Incremental with Active Full ONLY.
Any other tips for the DR6000's ?

I have 2 that are replicating between eachother over 10g link...
I have installed the Rapid CIFS driver on the Backup Server and Proxy servers and I am getting 800+mb/s ingestion.
Only issue I see now is getting the data back out onto 4x LTO5 tape drives. I have only been able to get a single 100mb/s stream coming off the device.
DR6000s have both 10G nic active and ESX hosts have 4x 10gb nics so no bottleneck there!

What is your egress (rehydration) rate?

Might have to use an intermediatary storage location:
So backup to Local Disks(7 restore points(1VBK+6VIB?)? 1 week) - Maybe less RP and use reverse incremental....
Then copy job of VBK and VIB to DR6000 daily, or "as new files appear" ? - Allows some sort of GFS retention....
Then tape job during the week - source is the local disk VBK + latest VIB rolled in(Virtual Synth Full).
So use the Local Disk copy for < 1 week restore point
I only ever read off the DR6000 in the case of a restore job of greater than a week ago and < a month
and greater than a month I use the tape copies.

Cheers.

[nikpolini]

Just had a look at the Backup Copy option, looks like it Injects the previous VIB into the VBK when the RP chain gets too long... not good for the DR6000...

[AdrianJM]

The Dell DR Series integration document for Veeam recommends that you don't use that option.
http://downloads.dell.com/Manuals/Commo ... _en-us.pdf
on page 19. Point 15.

"Note: Dell Recommends not to use Reversed incremental as this might have a negative impact on backup performance and savings."

Cheers

[foggy]

This is due to the fact that reverse incremental puts higher I/O load on the storage.

[mongie]

I have DR4100s. They work terribly with Veeam.

a) Dont try to write anything synthetic to them (primary backup)
b) No matter where you put your primary backups, when you use Backup Copy (GFS) to archive on the DR4100/DR6000, you're gonna have a bad time. The Merge/Health Check for some of my jobs (2-6TB Full) takes 2-3 days in most cases.

I'd suggest RDA support, but I doubt the DR series will exist once Dell and EMC merge.