Veeam very slow with Data Domain, need help to configure

[ro.Reen]

Hello,

I want to change from ARCserve to veeam, because we had some difficult issues with ARCserve in the last few years. My enviroment:

1. Source: 3 hosts with ESXi 5.5
   connected to an HP EVA by a FC (8GB/s)network
2. Path: The hosts are connected to the normal ethernet by 6x 1GB.
3. Target: I want to backup to an EMC DataDomain 620

So ok. I made 1 big Backup Job for around 25vms to get best dedup. Settings are like this:

• Incremental
• Active Full Backup on Saturday
• Inline dedup enabled
• Compression: Dedupe-friednly
• Optimize for LAN Target

Ok, I know the best way would be a physical machine with an FC card connected to the EVA. But I don't want to have additional physical machines, so I must handle that it may be slower cause of using 1GB LAN.

But: ATM Veeam makes these backups with just 950KB/s, so its needs nearly 5 hours for just 10GB of data. So I DON't think that the problem is just the LAN? Also there is so much time where just nothing happens? And no data is moved.
It also tells me a load like this: Source: 5-15% > Proxy 15-25% > Network 20-30% > Target around 70% (in sum more than 100%?)
Most times the target is the bottleneck.

If an incremental Job is done with 330Gb transfered it need more than 10 hours?

Any ideas?

Thanks...

[foggy]

But: ATM Veeam makes these backups with just 950KB/s, so its needs nearly 5 hours for just 10GB of data. So I DON't think that the problem is just the LAN? Also there is so much time where just nothing happens? And no data is moved. It also tells me a load like this: Source: 5-15% > Proxy 15-25% > Network 20-30% > Target around 70% (in sum more than 100%?)
Most times the target is the bottleneck.

Rene, that means that the write speed to the target storage is the bottleneck. How do you add DataDomain to Veeam B&R console (what type of backup repository do you use for it)?

Btw, there is a couple of existing threads regarding DataDomain performance issues and different ways of deployment, worth reviewing:

EMC Data Domain best practices
Veeam, DataDomain and Linux NFS share

[ro.Reen]

I added the DataDomain as CIFS Folder and set the limit for concurrent tasks to 4.

[jleblanc]

950KB/s tells me there is something wrong. When I was using CIFS to back up from Veeam to my Data Domain I was getting about 30-40 Mbs, then I moved over to using a Linux NFS proxy to write to my datadomain using NFS and get 100-120 Mbs. If you log into the datadomain support portal they have a section for Integration Guides, the Veeam to Datadomain integration guide has details on exactly how to configure this Linux NFS Proxy setup that I am using.

Also, you should check your job's dedplucation and compression settings to make sure you aren't compressing the data with veeam befor it writes to the data domain.

[GarionB]

Using DD4200 w/ 10GB interfaces, I have no issues achieving near line rate 700-800MBs over CIFS share. What you definitely want to do to ramp up your throughput is to break the jobs into multiple streams. Prior to me doing so I was only getting about 250MBs. NOTE: I do have inline dedupe DISABLED as recommended when target is DD; Compression = NONE and Optimized for local target.

1. create 3 more backup repositories ; ie DD1, DD2, DD3 all pointing to same CIFS share
2. Break job up into 3 or 4 jobs with max concurrent on each of 4 (assuming 16CPU box)

You will then get more concurrent streams to the DD box. Hope this helps

[ro.Reen]

Hello,

ok I changed the compression level to none and disabled the inline data deduplication. Now I get rates up to 6MB/s, but still the target is the bottlenet with around 70% to 98%, source is still under 5%, Proxy at around 10% to 20%, network under 10%.
Is there any way to check if everything is allright with the DataDomain?

Thanks!

[ro.Reen]

950KB/s tells me there is something wrong. When I was using CIFS to back up from Veeam to my Data Domain I was getting about 30-40 Mbs, then I moved over to using a Linux NFS proxy to write to my datadomain using NFS and get 100-120 Mbs. If you log into the datadomain support portal they have a section for Integration Guides, the Veeam to Datadomain integration guide has details on exactly how to configure this Linux NFS Proxy setup that I am using.

Could you please give me a link? I'm sorry, but the emc support portal is one of the worst things I've ever seen.

[foggy]

Here is how you could get it previously, not sure if it is still there.

[GarionB]

Hello,

ok I changed the compression level to none and disabled the inline data deduplication. Now I get rates up to 6MB/s, but still the target is the bottlenet with around 70% to 98%, source is still under 5%, Proxy at around 10% to 20%, network under 10%.
Is there any way to check if everything is allright with the DataDomain?

Thanks!

How many interfaces and what are the speed of your DD interfaces and have you verified the duplex settings

[ro.Reen]

There is one network interface connected with 1GB. Its not much, but when I watch the usage... it never goes over 30%.

I can also make 16 Streams in 1 Job, so why break it?

[ro.Reen]

There is no edit Button here? Duplex is set to full.

[tsightler]

The DD620 is quite an older unit at this point, what version of DD OS is it running? Any chance it happens to be pretty 5.x?

[GarionB]

There is one network interface connected with 1GB. Its not much, but when I watch the usage... it never goes over 30%.

I can also make 16 Streams in 1 Job, so why break it?

The multiple streams are made by having multiple jobs concurrently writing to the Data Domain. I am not referring to the concurrency you are seeing in the actual Veeam backup job. You must break up the jobs to get the desired affect. Just clone your existing job 3x and adjust the vms in each job accordingly.

[ro.Reen]

The DD620 is quite an older unit at this point, what version of DD OS is it running? Any chance it happens to be pretty 5.x?

Version 5.2.2.4-370754 is installed.

[ro.Reen]

Ok, so now I tried to make a backup to my EVA, just to test the speed. Its done with 173 MB/s. Bottleneck is Source with 95%.

So Network and Source shouldn't be the problem. Just the DataDomain is too slow.

[ro.Reen]

Just for your information: I've removed the e1000e network controller on the vm with veeam installed and installed the vmxnet3 network controller: Much faster now. Now I get up to 50MB/s. Some vms are still slower, but all in all much more performance!

[foggy]

Hm... while changing the proxy adapter itself could result in backup speed increase, I'm not sure how this can show up where target is the bottleneck.

[ro.Reen]

I don't know how this works, but this weekend I get 69MB/s with the full backup.
Bottlenecks:
Source: 6%
Proxy: 54%
Network: 22%
Target: 84%
So ok... need to speed up the target in any way... maybe using a LWL attached cheaper storage like qnap as target, which backups itself on DataDomain for longterm backup? Still looking for a solution, since I backup only to a storage in the same building.
Maybe direct backup to a fast lwl storage in backup windows and replicating to datadomain in another building?

[veremin]

Do you have any decommissioned physical server? If so, you provide it with bunch of directly attached drives and get one of the most fast and reliable targets at this price. Meanwhile, continue using DD appliance as the secondary target for long term archival purposes.

Thanks.