Veeam and Datadomain: reduce backup window

[cristiano.cumer]

Hello,

right now I have a backup window for my VM that spans from 9PM to 7AM. I would like to reduce it a little bit if possible.
Since my dd 670 appliance is the major bottleneck I would like to reduce IOs and data transferred to it.

I could use either the builtin veeam deduplication or the dedup friendly compression. My guess is that the veeam dedup would be more DD friendly than the compression, can anyone confirm it?
And I also have a question about veeam dedupe, it's done inline w/o aggravation of I/Os on the DD appliance or does it cause an increased IOs load on the target?

Cristiano

[Vitaliy S.]

Hello Cristiano,

I could use either the builtin veeam deduplication or the dedup friendly compression. My guess is that the veeam dedup would be more DD friendly than the compression, can anyone confirm it?

Yes, you're correct. Any compression level will be less friendly to a dedupe device, however our general recommendation is to use Veeam dedupe as well as the compression friendly mode while backing up to the dedupe device.

And I also have a question about veeam dedupe, it's done inline w/o aggravation of I/Os on the DD appliance or does it cause an increased IOs load on the target?

Veeam is using an in-line dedupe engine. Can you please tell me what backup mode (forward or reversed incremental) are you currently using for that backup job?

Thanks!

[cristiano.cumer]

Hello Cristiano,
Yes, you're correct. Any compression level will be less friendly to a dedupe device, however our general recommendation is to use Veeam dedupe as well as the compression friendly mode while backing up to the dedupe device.

Ok, I will try then with compression and dedup!

Veeam is using an in-line dedupe engine. Can you please tell me what backup mode (forward or reversed incremental) are you currently using for that backup job?

Thanks!

Forward, as it both reduces IOs and I need it for tapeout

Thanks

Cristiano

[foggy]

Just note that while it will decrease the time required to write data to the target storage, it will, in its turn, require additional time to compress and dedupe data.

[cristiano.cumer]

Just note that while it will decrease the time required to write data to the target storage, it will, in its turn, require additional time to compress and dedupe data.

Hello Foggy,

I will run some tests and try to find the sweet spot!

Cristiano

[jleblanc]

Hi All, Im preparing my jobs to start backing up to an EMC DD 620 and I understand that best practices dictate not to use Veeam compression with dedupe appliances. However, I notice there is a 'dedupe friendly' compression configuration. Should I be using No compression or Dedupe friendly compression with the Data Domain appliances? Thanks!

[Tobias_Elfstrom]

There are recommendations and yes, you will have to evaluate this if your particular environment.

But as a rule sending the data to your dedupe device uncompressed and undeduped will probably give you a better overall disk usage ratio but will also most probably give you a longer backup windows (more data to transfer). Turning on compression and dedupe within Veeam will give you less data to transfer and will make your dedupe device use more CPU do be able to find stuff to dedupe and compress. Finding the sweetspot will depend on your VM's and the demands you have so you will have to try around a bit.

BR Tobias.

[dellock6]

Guys, remember that DataDomain and other dedup appliances has inline deduplication, so the data reduction happens while data are ingested.
So, a completely uncompressed backup is indeed going to extend backup duration. A "dedupe friendly" setting is a good compromise to start with.

Luca.

[mongie]

Wouldn't it be nice if Veeam supported DD Boost / Catalyst.

Oh I long for the day.

[Tobias_Elfstrom]

Wouldn't it be nice if Veeam supported DD Boost / Catalyst.

That it would indeed
I would guess though that the demand for such thing is relatively small..

BR Tobias

[Gostev]

We looked at this and apparently DDBoost and Catalyst are simply not designed for B&R I/O patterns... they are more for legacy, "streaming" backup applications.

[ChristinaPrimeauEMC]

Hi Christiano,

I know its been about a month since the last post in this thread, but I wanted to check in and see how everything is going for you. I was really surprised to see that you were noting the DD670 as your bottleneck for backup performance as it is designed to push 3.1TBs of data/hour. Can I ask how much data you are trying to backup?

As far as activating Veeam de-duplication or compression, this is highly unadvised from the EMC team. To use Data Domain as the backend for Veeam, you must not use compression or any dedupe from Veeam (turn them off). Rely on DD to do the deduplication and feed it a regular backup stream. If you use Veeam compression or dedupe capabilities, the stream will have a very low likelihood of being further reduplicated because all the segments will look different to DD. It would be similar to turning on encryption before sending to DD, although not quite that bad. If you did use the Veeam dedupe, DD would still deduplicate further, but not at the typical 20-30x we normally see. In this scenario, the payback is slowed down because the dedupe ratio is reduced and the sizing is off.

As Mongie suggested, DD Boost would be an excelent way to speed up your backup window but is not currently supported by Veeam.

Please feel free to reach out if there is anywhere I can asist.

Christina
christina.primeau@emc.com

[dellock6]

Christina, that number means 861 MB per second, are you sure? I've seen it also in the website (3.6 there to be honest) but really almost 1 GB per second sounds to me like an incredible number to say the least. that is a "best case" in which conditions? A stream coming from a database servers where you can reach high dedup ration as soon as data arrives?
Do you have some "from the field" number in situations with VMware beeing saved by Veeam?

Sorry for jumping into the thread, but those numbers made me curious.

Luca.

[Helqasem]

As far as activating Veeam de-duplication or compression, this is highly unadvised from the EMC team. To use Data Domain as the backend for Veeam, you must not use compression or any dedupe from Veeam (turn them off). Rely on DD to do the deduplication and feed it a regular backup stream. If you use Veeam compression or dedupe capabilities, the stream will have a very low likelihood of being further reduplicated because all the segments will look different to DD. It would be similar to turning on encryption before sending to DD, although not quite that bad. If you did use the Veeam dedupe, DD would still deduplicate further, but not at the typical 20-30x we normally see. In this scenario, the payback is slowed down because the dedupe ratio is reduced and the sizing is off.

Disagree. I have deployed Veeam alongside DD many times and the configuration should be tuned on a per system basis.

My recommendation would be to test by initially have Veeam dedupe and compression switched off, this will is most cases put the bottleneck at the DD.

You should then work to bring load off the DD by moving some of the processing to the Veeam proxy, initially testing by increasing dedupe and further testing with low levels of compression. You should continue to increase Veeam's processing to a point that you are happy with the performance and relative space usage. I can cite one environment I have optimzed in this way where able to take 6TB of source data and using both Veeam and DD was able to reduce this down to 120GB on the DD.

In addition to moving load off the DD, what happens when you need to take data off the DD for long term retention (tape, cloud storage, etc)? If you are only using DD dedupe the data would need to be completely rehydrated before moving off the device. If Veeam dedupe and compression are used the backup files stay portable. DD Boost doesn't help with this scenario. Most customers want their backup data to be portable, which is why Veeam is a great solution to prevent vendor lock-in.. we're storage agnostic.

[lti]

Hi,

Very interesting discussion.
We are using Veeam with DataDomain, also a DD670. We reach a continuous rate of 400 MB/s during backups with peaks at 450-500 MB/s. We have no problem with backup window, even with ~70 Jobs configured (~300VMs). We have about 180TB of Veeam backups stored on the DD670 with no more than 1 month retention.
Usually, the bottleneck is not the destination, but the source (SAN disks not fast enough).
Proxy don't run more than 4 jobs in parallel, and we disabled compression/deduplication into Veeam. Everything is done by the DataDomain.
We even increased the compression ratio on the DataDomain (from lz to gzfast) without major impact (Only higher cpu usage on, datadomain)
We are using physical proxy server, with direct SAN access and 10Gb/s fiber connectivity
Data is replicated continuously to another dd670 offsite.

Major advantage of DataDomain is to use Global Deduplication, so we don't care about per-job deduplication. And of course replication
Disadvantage is poor random I/O so we can not use features like synthetic fulls or instant vm recovery

I think you have an issue with your configuration, can you give us more details about it ?

[lgomez]

lti what are you guys using to dump your backups NFS or CIFFS? We have a major problem with NFS being really slow however CIFS perform without no problems.

[foggy]

Here is another existing topic that might be applicable to your case, please review: Veeam, DataDomain and Linux NFS share.

[lti]

lti what are you guys using to dump your backups NFS or CIFFS? We have a major problem with NFS being really slow however CIFS perform without no problems.

We are using CIFS
During the design, we planned to use Linux NFS proxies, but we never installed them, as the performance with CIFS was good

[chrisdearden]

As I understand it , in more recent version of the DD OS - the delta between CIFS and NFS performance has been considerably reduced to the point where its not really worth deploying the extra linux repository.

[soylent]

What about the "Deduplication Storage Compatibility" options when adding a Data Domain as a repository in Veeam?

There are two options in the advanced menu: "Align backup file data blocks" and "Decompress backup data blocks before storing". Wouldn't enabling these options change the way the job's compression works? So that you can enable compression on the job to reduce network traffic, but it will write to the storage uncompressed?

[Gostev]

You definitely should not be using the align option, as description explains, this one is not intended for use with DataDomain deduplication engine, and will reduce the deduplication ratio slightly if enabled.

As for enabling decompression before storing, there are two opinions. DataDomain will recommend that you enable this option, because this will result in better deduplication on their side (essentially, the quest for bigger numbers in their reports). Veeam will recommend that instead, you use dedupe-friendly compression in the job (this significantly reduces backup window at only slight impact on effective data reduction ratio). The truth is somewhere in between, as always - and depends on where you are standing with your backup window today.

[emachabert]

From what I have seen for the last two years on both DD and HP storeonce is that enabling Veeam inline deduplication has no impact on the backup speed, but is has on read spead and read latency.
Even if those appliances are not good at all (and designed for by the way...) on random reads, try to run an instant recovery from the Appliance using both configuration (with and without inline dedupe).
In all my tests there was a noticeable difference in read latency, with inline dedupe enabled it resulted in super poor performance with all jobs using vPower.