Comprehensive data protection for all workloads
Post Reply
cristiano.cumer
Enthusiast
Posts: 34
Liked: 9 times
Joined: Nov 23, 2011 11:18 pm
Full Name: Cristianno Cumer
Contact:

Veeam and Datadomain: reduce backup window

Post by cristiano.cumer »

Hello,

right now I have a backup window for my VM that spans from 9PM to 7AM. I would like to reduce it a little bit if possible.
Since my dd 670 appliance is the major bottleneck I would like to reduce IOs and data transferred to it.

I could use either the builtin veeam deduplication or the dedup friendly compression. My guess is that the veeam dedup would be more DD friendly than the compression, can anyone confirm it?
And I also have a question about veeam dedupe, it's done inline w/o aggravation of I/Os on the DD appliance or does it cause an increased IOs load on the target?

Cristiano
Vitaliy S.
VP, Product Management
Posts: 27377
Liked: 2800 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Veeam and Datadomain: reduce backup window

Post by Vitaliy S. »

Hello Cristiano,
cristiano.cumer wrote:I could use either the builtin veeam deduplication or the dedup friendly compression. My guess is that the veeam dedup would be more DD friendly than the compression, can anyone confirm it?
Yes, you're correct. Any compression level will be less friendly to a dedupe device, however our general recommendation is to use Veeam dedupe as well as the compression friendly mode while backing up to the dedupe device.
cristiano.cumer wrote:And I also have a question about veeam dedupe, it's done inline w/o aggravation of I/Os on the DD appliance or does it cause an increased IOs load on the target?
Veeam is using an in-line dedupe engine. Can you please tell me what backup mode (forward or reversed incremental) are you currently using for that backup job?

Thanks!
cristiano.cumer
Enthusiast
Posts: 34
Liked: 9 times
Joined: Nov 23, 2011 11:18 pm
Full Name: Cristianno Cumer
Contact:

Re: Veeam and Datadomain: reduce backup window

Post by cristiano.cumer » 1 person likes this post

Vitaliy S. wrote:Hello Cristiano,
Yes, you're correct. Any compression level will be less friendly to a dedupe device, however our general recommendation is to use Veeam dedupe as well as the compression friendly mode while backing up to the dedupe device.
Ok, I will try then with compression and dedup!
Vitaliy S. wrote: Veeam is using an in-line dedupe engine. Can you please tell me what backup mode (forward or reversed incremental) are you currently using for that backup job?

Thanks!
Forward, as it both reduces IOs and I need it for tapeout

Thanks

Cristiano
foggy
Veeam Software
Posts: 21139
Liked: 2141 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Veeam and Datadomain: reduce backup window

Post by foggy »

Just note that while it will decrease the time required to write data to the target storage, it will, in its turn, require additional time to compress and dedupe data.
cristiano.cumer
Enthusiast
Posts: 34
Liked: 9 times
Joined: Nov 23, 2011 11:18 pm
Full Name: Cristianno Cumer
Contact:

Re: Veeam and Datadomain: reduce backup window

Post by cristiano.cumer »

foggy wrote:Just note that while it will decrease the time required to write data to the target storage, it will, in its turn, require additional time to compress and dedupe data.

Hello Foggy,

I will run some tests and try to find the sweet spot!

Cristiano
jleblanc
Influencer
Posts: 12
Liked: 1 time
Joined: Apr 26, 2013 4:44 pm
Full Name: Jeff LEBlanc
Contact:

[MERGED] Yet Another Veeam and Data Domain Thread

Post by jleblanc »

Hi All, Im preparing my jobs to start backing up to an EMC DD 620 and I understand that best practices dictate not to use Veeam compression with dedupe appliances. However, I notice there is a 'dedupe friendly' compression configuration. Should I be using No compression or Dedupe friendly compression with the Data Domain appliances? Thanks!
Tobias_Elfstrom
Enthusiast
Posts: 84
Liked: 8 times
Joined: Jul 04, 2012 6:32 am
Full Name: Tobias Elfstrom
Contact:

Re: Veeam and Datadomain: reduce backup window

Post by Tobias_Elfstrom »

There are recommendations and yes, you will have to evaluate this if your particular environment.

But as a rule sending the data to your dedupe device uncompressed and undeduped will probably give you a better overall disk usage ratio but will also most probably give you a longer backup windows (more data to transfer). Turning on compression and dedupe within Veeam will give you less data to transfer and will make your dedupe device use more CPU do be able to find stuff to dedupe and compress. Finding the sweetspot will depend on your VM's and the demands you have so you will have to try around a bit.

BR Tobias.
dellock6
VeeaMVP
Posts: 6166
Liked: 1971 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Veeam and Datadomain: reduce backup window

Post by dellock6 » 1 person likes this post

Guys, remember that DataDomain and other dedup appliances has inline deduplication, so the data reduction happens while data are ingested.
So, a completely uncompressed backup is indeed going to extend backup duration. A "dedupe friendly" setting is a good compromise to start with.

Luca.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
mongie
Expert
Posts: 152
Liked: 24 times
Joined: May 16, 2011 4:00 am
Full Name: Alex Macaronis
Location: Brisbane, Australia
Contact:

Re: Veeam and Datadomain: reduce backup window

Post by mongie »

Wouldn't it be nice if Veeam supported DD Boost / Catalyst.

Oh I long for the day.
Tobias_Elfstrom
Enthusiast
Posts: 84
Liked: 8 times
Joined: Jul 04, 2012 6:32 am
Full Name: Tobias Elfstrom
Contact:

Re: Veeam and Datadomain: reduce backup window

Post by Tobias_Elfstrom »

mongie wrote:Wouldn't it be nice if Veeam supported DD Boost / Catalyst.
That it would indeed :)
I would guess though that the demand for such thing is relatively small.. :|

BR Tobias
Gostev
Chief Product Officer
Posts: 31814
Liked: 7302 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Veeam and Datadomain: reduce backup window

Post by Gostev »

We looked at this and apparently DDBoost and Catalyst are simply not designed for B&R I/O patterns... they are more for legacy, "streaming" backup applications.
ChristinaPrimeauEMC
Lurker
Posts: 1
Liked: never
Joined: Jun 06, 2013 7:02 pm
Full Name: Christina Primeau
Contact:

Re: Veeam and Datadomain: reduce backup window

Post by ChristinaPrimeauEMC »

Hi Christiano,

I know its been about a month since the last post in this thread, but I wanted to check in and see how everything is going for you. I was really surprised to see that you were noting the DD670 as your bottleneck for backup performance as it is designed to push 3.1TBs of data/hour. Can I ask how much data you are trying to backup?

As far as activating Veeam de-duplication or compression, this is highly unadvised from the EMC team. To use Data Domain as the backend for Veeam, you must not use compression or any dedupe from Veeam (turn them off). Rely on DD to do the deduplication and feed it a regular backup stream. If you use Veeam compression or dedupe capabilities, the stream will have a very low likelihood of being further reduplicated because all the segments will look different to DD. It would be similar to turning on encryption before sending to DD, although not quite that bad. If you did use the Veeam dedupe, DD would still deduplicate further, but not at the typical 20-30x we normally see. In this scenario, the payback is slowed down because the dedupe ratio is reduced and the sizing is off.

As Mongie suggested, DD Boost would be an excelent way to speed up your backup window but is not currently supported by Veeam.

Please feel free to reach out if there is anywhere I can asist.

Christina
christina.primeau@emc.com
dellock6
VeeaMVP
Posts: 6166
Liked: 1971 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Veeam and Datadomain: reduce backup window

Post by dellock6 »

Christina, that number means 861 MB per second, are you sure? I've seen it also in the website (3.6 there to be honest) but really almost 1 GB per second sounds to me like an incredible number to say the least. that is a "best case" in which conditions? A stream coming from a database servers where you can reach high dedup ration as soon as data arrives?
Do you have some "from the field" number in situations with VMware beeing saved by Veeam?

Sorry for jumping into the thread, but those numbers made me curious.

Luca.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
Helqasem
Expert
Posts: 226
Liked: 28 times
Joined: Jan 27, 2012 11:31 am
Full Name: Hani El-Qasem
Contact:

Re: Veeam and Datadomain: reduce backup window

Post by Helqasem » 1 person likes this post

ChristinaPrimeauEMC wrote: As far as activating Veeam de-duplication or compression, this is highly unadvised from the EMC team. To use Data Domain as the backend for Veeam, you must not use compression or any dedupe from Veeam (turn them off). Rely on DD to do the deduplication and feed it a regular backup stream. If you use Veeam compression or dedupe capabilities, the stream will have a very low likelihood of being further reduplicated because all the segments will look different to DD. It would be similar to turning on encryption before sending to DD, although not quite that bad. If you did use the Veeam dedupe, DD would still deduplicate further, but not at the typical 20-30x we normally see. In this scenario, the payback is slowed down because the dedupe ratio is reduced and the sizing is off.
Disagree. I have deployed Veeam alongside DD many times and the configuration should be tuned on a per system basis.

My recommendation would be to test by initially have Veeam dedupe and compression switched off, this will is most cases put the bottleneck at the DD.

You should then work to bring load off the DD by moving some of the processing to the Veeam proxy, initially testing by increasing dedupe and further testing with low levels of compression. You should continue to increase Veeam's processing to a point that you are happy with the performance and relative space usage. I can cite one environment I have optimzed in this way where able to take 6TB of source data and using both Veeam and DD was able to reduce this down to 120GB on the DD.

In addition to moving load off the DD, what happens when you need to take data off the DD for long term retention (tape, cloud storage, etc)? If you are only using DD dedupe the data would need to be completely rehydrated before moving off the device. If Veeam dedupe and compression are used the backup files stay portable. DD Boost doesn't help with this scenario. Most customers want their backup data to be portable, which is why Veeam is a great solution to prevent vendor lock-in.. we're storage agnostic.
lti
Novice
Posts: 6
Liked: never
Joined: Sep 17, 2012 5:40 pm
Contact:

Re: Veeam and Datadomain: reduce backup window

Post by lti »

Hi,

Very interesting discussion.
We are using Veeam with DataDomain, also a DD670. We reach a continuous rate of 400 MB/s during backups with peaks at 450-500 MB/s. We have no problem with backup window, even with ~70 Jobs configured (~300VMs). We have about 180TB of Veeam backups stored on the DD670 with no more than 1 month retention.
Usually, the bottleneck is not the destination, but the source (SAN disks not fast enough).
Proxy don't run more than 4 jobs in parallel, and we disabled compression/deduplication into Veeam. Everything is done by the DataDomain.
We even increased the compression ratio on the DataDomain (from lz to gzfast) without major impact (Only higher cpu usage on, datadomain)
We are using physical proxy server, with direct SAN access and 10Gb/s fiber connectivity
Data is replicated continuously to another dd670 offsite.

Major advantage of DataDomain is to use Global Deduplication, so we don't care about per-job deduplication. And of course replication
Disadvantage is poor random I/O so we can not use features like synthetic fulls or instant vm recovery

I think you have an issue with your configuration, can you give us more details about it ?
lgomez
Lurker
Posts: 2
Liked: 1 time
Joined: Jun 26, 2013 6:27 pm
Contact:

Re: Veeam and Datadomain: reduce backup window

Post by lgomez »

lti what are you guys using to dump your backups NFS or CIFFS? We have a major problem with NFS being really slow however CIFS perform without no problems.
foggy
Veeam Software
Posts: 21139
Liked: 2141 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Veeam and Datadomain: reduce backup window

Post by foggy »

Here is another existing topic that might be applicable to your case, please review: Veeam, DataDomain and Linux NFS share.
lti
Novice
Posts: 6
Liked: never
Joined: Sep 17, 2012 5:40 pm
Contact:

Re: Veeam and Datadomain: reduce backup window

Post by lti »

lgomez wrote:lti what are you guys using to dump your backups NFS or CIFFS? We have a major problem with NFS being really slow however CIFS perform without no problems.
We are using CIFS
During the design, we planned to use Linux NFS proxies, but we never installed them, as the performance with CIFS was good
chrisdearden
Veteran
Posts: 1531
Liked: 226 times
Joined: Jul 21, 2010 9:47 am
Full Name: Chris Dearden
Contact:

Re: Veeam and Datadomain: reduce backup window

Post by chrisdearden »

As I understand it , in more recent version of the DD OS - the delta between CIFS and NFS performance has been considerably reduced to the point where its not really worth deploying the extra linux repository.
soylent
Enthusiast
Posts: 61
Liked: 7 times
Joined: Aug 01, 2012 8:33 pm
Full Name: Max
Location: Fort Lauderdale, Florida
Contact:

Re: Veeam and Datadomain: reduce backup window

Post by soylent »

What about the "Deduplication Storage Compatibility" options when adding a Data Domain as a repository in Veeam?

There are two options in the advanced menu: "Align backup file data blocks" and "Decompress backup data blocks before storing". Wouldn't enabling these options change the way the job's compression works? So that you can enable compression on the job to reduce network traffic, but it will write to the storage uncompressed?
Gostev
Chief Product Officer
Posts: 31814
Liked: 7302 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Veeam and Datadomain: reduce backup window

Post by Gostev »

You definitely should not be using the align option, as description explains, this one is not intended for use with DataDomain deduplication engine, and will reduce the deduplication ratio slightly if enabled.

As for enabling decompression before storing, there are two opinions. DataDomain will recommend that you enable this option, because this will result in better deduplication on their side (essentially, the quest for bigger numbers in their reports). Veeam will recommend that instead, you use dedupe-friendly compression in the job (this significantly reduces backup window at only slight impact on effective data reduction ratio). The truth is somewhere in between, as always - and depends on where you are standing with your backup window today.
emachabert
Veeam Vanguard
Posts: 395
Liked: 169 times
Joined: Nov 17, 2010 11:42 am
Full Name: Eric Machabert
Location: France
Contact:

Re: Veeam and Datadomain: reduce backup window

Post by emachabert »

From what I have seen for the last two years on both DD and HP storeonce is that enabling Veeam inline deduplication has no impact on the backup speed, but is has on read spead and read latency.
Even if those appliances are not good at all (and designed for by the way...) on random reads, try to run an instant recovery from the Appliance using both configuration (with and without inline dedupe).
In all my tests there was a noticeable difference in read latency, with inline dedupe enabled it resulted in super poor performance with all jobs using vPower.
Veeamizing your IT since 2009/ Veeam Vanguard 2015 - 2023
Post Reply

Who is online

Users browsing this forum: No registered users and 54 guests