Backup slow with EMC DD2500 and DDBoost

[rereduck]

Hello,

Case ID : 00720532

We have just acquire an EMC DD2500 with DDBoost and Veeam V8 (with Patch 1 since yesterday) and backups are very slow.

Before DD2500 we backup our VM with Veeam V7 and V8 on local disk on a physical server with only one proxy (the physical server).

With this configuration, we managed to daily backup 10TB in incremental mode for 70/80 vm in 3h to 6h.Bottleneck was Source (99%/16%/2%/0%) and processing rate was approximativly 60MB/s.

With DD2500 and DDBoost, we need between 15 and 28 hours to backup 10TB in incremental mode.Bottleneck is Network (50%/11%/64%/41%) and processing rate is approximativly 20MB/s.

Do you think is it normal ? Is it a configuration issue ? How can I reduce the backup Window ?

To sump up :

Before :
One Physcial Server with 30Tb disk
One Proxy (the psysical server)
2x 10GB/s for the network.
One Veeam Job with 70VM and 10TB in daily incremental and Synthetic full once a week.
Compression leve : High
Storage Optimization : Local Target
Inlinde dedup : Enabled
Time to backup : between 3h and 6h


Now :
DD2500 with 45TB
4x 1GB/s for the network (We plan to migrate to 2x10Gb/s)
Two Proxy (The Physical server and one VM)
One Veeam Job with 70VM and 10TB in daily incremental and Synthetic full once a week.
Compression leve : Optimal
Storage Optimization : LAN Target
Inlinde dedup : Enabled
Time to backup : between 15h and 28h.

Virtual infrastructure :
Vmware Vsphere 5.5

Thanks for your help

[Gostev]

I would have support start from investigating "network" being a bottleneck on 4Gb Ethernet with mere 20MB/s processing rate, as this looks extremely suspicious. Your current bottleneck stats basically mean that the job spends most time waiting for the data to be transferred over network... backup target type does not even matter in such conditions. Thanks!

[rereduck]

Thanks Gostev for your answer.

I've read that Veeam use only one network link to transfer data (because there only one Job) so only 1Gb in my case that's why we have planned to migrate from 4x 1Gb to 2x10Gb but I was not sure thaht it was a network issue.

So I think I'll wait for 2x10Gb to see if it is better.

Thanks

[Gostev]

Not if you actually teamed the network links in Windows, in which case it is OS itself that will leverage all 4 links for bandwidth aggregation. You are right that Veeam by itself cannot team multiple links, but we don't implement this simply because teaming is normally done on lower OSI stack level.

[tsightler]

And Veeam does use mulitple TCP connections, so assuming the load balancing used by the team is based on TCP port then you should be able to leverage mulitple links. That being said, you're talking about speeds of 20MB/s here, that's not even close to a single 1GbE link so clearly something isn't right with the dataflow.

[rereduck]

To complete my first post, here are screenshot of the same virtual machine with DD2500 and with our Physical Server and Local Disks :

[rereduck]

And Veeam does use mulitple TCP connections, so assuming the load balancing used by the team is based on TCP port then you should be able to leverage mulitple links. That being said, you're talking about speeds of 20MB/s here, that's not even close to a single 1GbE link so clearly something isn't right with the dataflow.

Sorry but why do you say that 20MB/s is not even close to a 1GbE link ? 1GbE link max speed is not 125MB/S ?

Thanks for your help tsightler

[Gostev]

Because 20 MB/s is not even close to 125 MB/s?

[chrisdearden]

when you added the DD - which server did you run the boost gateway on?

[Guyzmo]

You need to disable compression and inline dedup. (Best practice for DD repository)

Maybe this settings "enable" can reduce the speed.

[tsightler]

Because 20 MB/s is not even close to 125 MB/s?

Exactly. The problem here is obviously not bandwidth, at least not directly. It may be dataflow not optimally using the data paths that you have, but with 1GbE you should be able to get far higher than even your previous speeds. Honestly, your original speeds look more like 100Mb speeds that 1Gb.

You mention that you are using a physical backup server, but I see at least two proxies in use, and one is using NBD mode (the screen was cut off so I couldn't see the other). Where is "VeeamProxy" located in relation to "VMware Backup Proxy"? And as previously asked, which system did you select as the gateway server to to DD? It looks like this new job is for some reason using VeeamProxy via NBD and that this server might be across a slow link. If it's also being used as the DD gateway then this is the problem and also explain why network is the bottleneck.

As a temporary troubleshooting step I'd suggest manually selecting the proxy for that job to use only the "VMware Backup Proxy" and see if that helps.

[meilicke]

Just as a data point, we have:

* Veeam v8 Patch 1
* DD2500 head plus tray used as a boost repository. All jobs go here.
* 110 VMs, 12.8TB over four jobs.
* Between 115MB/s and 223MB/s processing rate last night.
* The total time required is about an 1 hour and a half. We are using boost repositories.
* The DD reports a throughput of around
* The source is our bottleneck.
* The DD is connected via two 1G links, not 10G.

[rereduck]

Because 20 MB/s is not even close to 125 MB/s?

Oh ok ^^ , sorry I did not understand previous post from tsightler

when you added the DD - which server did you run the boost gateway on?

Automatic Selection

Exactly. The problem here is obviously not bandwidth, at least not directly. It may be dataflow not optimally using the data paths that you have, but with 1GbE you should be able to get far higher than even your previous speeds. Honestly, your original speeds look more like 100Mb speeds that 1Gb.

You mention that you are using a physical backup server, but I see at least two proxies in use, and one is using NBD mode (the screen was cut off so I couldn't see the other). Where is "VeeamProxy" located in relation to "VMware Backup Proxy"? And as previously asked, which system did you select as the gateway server to to DD? It looks like this new job is for some reason using VeeamProxy via NBD and that this server might be across a slow link. If it's also being used as the DD gateway then this is the problem and also explain why network is the bottleneck.

As a temporary troubleshooting step I'd suggest manually selecting the proxy for that job to use only the "VMware Backup Proxy" and see if that helps.

At first we only use one physical server as proxy but trying to improve performance I have add a second virtual proxy server to see if it is better with no results so I think I'll turn off virtual proxy.

You need to disable compression and inline dedup. (Best practice for DD repository)
Maybe this settings "enable" can reduce the speed.

Thanks Guyzmo, I think you have solved my problem With one job, speed is now arround 60MB/s and I backup 10,4TB in 8 hours instead of 20 hours ! I have followed recommandation in this KB : http://www.veeam.com/kb1745 (Why I didn't see it before ? ^^) !

I think I'll create 4 jobs instead of 1 like Meilicke to improve again performances. Thank to you for your feedback Meilicke

If you have any other advises to improve again performances, I'm interested

[Guyzmo]

Hi

If you have followed the kb you have configure compression with dedupe friendly.

With ddboost, I think the best configuration for compression is None.
Can you test it and give us the speed?

Thanks.

[SGalbincea]

I am interested to know which works best as well. Currently we use dedupe friendly and get decent rates, but I may switch to none to see what happens.

[Gostev]

I have followed recommandation in this KB : http://www.veeam.com/kb1745

You may not have noticed, but this KB applies to versions 6.x, 7.x only.
B&R v8 and Data Domain users should refer to KB1956 instead.

[emachabert]

Is everything ok regarding network cabling ? Duplex and speed ? At both ends.
Regarding your setup (proxy using 10gb/s and DD using gbe), I would suspect pause frames (flow control) causing the slow down on the network path.
Is the proxy network graph looking like a crenel ?
Did you do a network capture to look for pause frame or massive TCP retransmission ?

[rereduck]

Hi

If you have followed the kb you have configure compression with dedupe friendly.

With ddboost, I think the best configuration for compression is None.
Can you test it and give us the speed?

Thanks.

I'll do it next week and I'll give you feedback about that.

You may not have noticed, but this KB applies to versions 6.x, 7.x only.
B&R v8 and Data Domain users should refer to KB1956 instead.

Thanks Gostev but in this KB is not about job settings and it says :

"For more information regarding Backup Job settings please reference:
http://www.veeam.com/kb1745"

Do you have any best practices for job settings and DataDomain update to version 8 ?

Thanks.

[rereduck]

Is everything ok regarding network cabling ? Duplex and speed ? At both ends.
Regarding your setup (proxy using 10gb/s and DD using gbe), I would suspect pause frames (flow control) causing the slow down on the network path.
Is the proxy network graph looking like a crenel ?
Did you do a network capture to look for pause frame or massive TCP retransmission ?

Since I've set "compression mode " to "dedup friendly" and disable inline dedup in job settings, performance are much better and network is not a problem.

I'm always looking to reduce my backup time but I think I'll create 4 jobs instead of one to do that.

Thanks for your help.

[Gostev]

Do you have any best practices for job settings and DataDomain update to version 8?

v8 sets the recommended job settings automatically depending on the backup repository you chose for the job, so you no longer have to worry about that part.

[SGalbincea]

That sort of happens. When I click next it prompts to set the advanced settings, and it does indeed uncheck inline dedupe...but compression is set to optimal, not dedupe friendly or none which I understand are the best settings for DataDomain with DDBoost.

[Guyzmo]

I think None is the best setting for speed but Dedupe Friendly is ok.

Could you test with Compression to None and compare speeds?

[Gostev]

compression is set to optimal, not dedupe friendly or none which I understand are the best settings for DataDomain with DDBoost

That is the correct statement.

Data Domain requires NO compression, which is why that advanced backup repository option to uncompress before saving is checked by default for DDBoost repositories. Whereas having Optimal compression enabled in the job reduces network bandwidth usage without any side effects.

Could you test with Compression to None and compare speeds?

Does not matter, as EMC requires that any sort of compression is disabled.

[rereduck]

Hello, me again

I see no big difference between compression set to "none" and "dedup friendly".

I'm a trying to do 3 jobs instead of 1 to speed up backup time but since I've do that Veeam Backup server did not respond and freeze and backup failed.

I can see this error on Veeam Backup :

Failed to wait mutex ... :



On my jobs I can see this error on all my VMs :

12/01/2015 20:01:22 :: Processing Squid Error: Microsoft SQL server hosting the configuration database is currently unavailable. Possible reasons are heavy load, networking issue, server reboot, or hot backup.
Please wait, and try again later.

Do you have any ideas why my server is freezing ?

My SQL Veeam Database is a SQL Server Express 2005 edition, can it be a problem ?

Here my infrastructure :

Veeam Backup Server and Proxy : 1 Physical Server with 2x10Gbe (SQL Express 2005, 12 core (24 logical) and 16G ram) in LAN Mode (no Direct SAN)
Veeam Proxy : 1 VM (8 core and 8G ram)
Repository : 1 DataDomain DD2500 connected with 2x10Gbe

Vmware :
80 VM and 10TB

Thanks