Comprehensive data protection for all workloads
Post Reply
rereduck
Enthusiast
Posts: 28
Liked: 1 time
Joined: Dec 09, 2013 9:11 am
Contact:

Backup slow with EMC DD2500 and DDBoost

Post by rereduck »

Hello,

Case ID : 00720532

We have just acquire an EMC DD2500 with DDBoost and Veeam V8 (with Patch 1 since yesterday) and backups are very slow.

Before DD2500 we backup our VM with Veeam V7 and V8 on local disk on a physical server with only one proxy (the physical server).

With this configuration, we managed to daily backup 10TB in incremental mode for 70/80 vm in 3h to 6h.Bottleneck was Source (99%/16%/2%/0%) and processing rate was approximativly 60MB/s.

With DD2500 and DDBoost, we need between 15 and 28 hours to backup 10TB in incremental mode.Bottleneck is Network (50%/11%/64%/41%) and processing rate is approximativly 20MB/s.

Do you think is it normal ? Is it a configuration issue ? How can I reduce the backup Window ?

To sump up :

Before :
One Physcial Server with 30Tb disk
One Proxy (the psysical server)
2x 10GB/s for the network.
One Veeam Job with 70VM and 10TB in daily incremental and Synthetic full once a week.
Compression leve : High
Storage Optimization : Local Target
Inlinde dedup : Enabled
Time to backup : between 3h and 6h


Now :
DD2500 with 45TB
4x 1GB/s for the network (We plan to migrate to 2x10Gb/s)
Two Proxy (The Physical server and one VM)
One Veeam Job with 70VM and 10TB in daily incremental and Synthetic full once a week.
Compression leve : Optimal
Storage Optimization : LAN Target
Inlinde dedup : Enabled
Time to backup : between 15h and 28h.

Virtual infrastructure :
Vmware Vsphere 5.5

Thanks for your help
Gostev
Chief Product Officer
Posts: 31816
Liked: 7302 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Backup slow with EMC DD2500 and DDBoost

Post by Gostev »

I would have support start from investigating "network" being a bottleneck on 4Gb Ethernet with mere 20MB/s processing rate, as this looks extremely suspicious. Your current bottleneck stats basically mean that the job spends most time waiting for the data to be transferred over network... backup target type does not even matter in such conditions. Thanks!
rereduck
Enthusiast
Posts: 28
Liked: 1 time
Joined: Dec 09, 2013 9:11 am
Contact:

Re: Backup slow with EMC DD2500 and DDBoost

Post by rereduck »

Thanks Gostev for your answer.

I've read that Veeam use only one network link to transfer data (because there only one Job) so only 1Gb in my case that's why we have planned to migrate from 4x 1Gb to 2x10Gb but I was not sure thaht it was a network issue.

So I think I'll wait for 2x10Gb to see if it is better.

Thanks
Gostev
Chief Product Officer
Posts: 31816
Liked: 7302 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Backup slow with EMC DD2500 and DDBoost

Post by Gostev »

Not if you actually teamed the network links in Windows, in which case it is OS itself that will leverage all 4 links for bandwidth aggregation. You are right that Veeam by itself cannot team multiple links, but we don't implement this simply because teaming is normally done on lower OSI stack level.
tsightler
VP, Product Management
Posts: 6035
Liked: 2860 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Backup slow with EMC DD2500 and DDBoost

Post by tsightler »

And Veeam does use mulitple TCP connections, so assuming the load balancing used by the team is based on TCP port then you should be able to leverage mulitple links. That being said, you're talking about speeds of 20MB/s here, that's not even close to a single 1GbE link so clearly something isn't right with the dataflow.
rereduck
Enthusiast
Posts: 28
Liked: 1 time
Joined: Dec 09, 2013 9:11 am
Contact:

Re: Backup slow with EMC DD2500 and DDBoost

Post by rereduck »

To complete my first post, here are screenshot of the same virtual machine with DD2500 and with our Physical Server and Local Disks :

Image
rereduck
Enthusiast
Posts: 28
Liked: 1 time
Joined: Dec 09, 2013 9:11 am
Contact:

Re: Backup slow with EMC DD2500 and DDBoost

Post by rereduck »

tsightler wrote:And Veeam does use mulitple TCP connections, so assuming the load balancing used by the team is based on TCP port then you should be able to leverage mulitple links. That being said, you're talking about speeds of 20MB/s here, that's not even close to a single 1GbE link so clearly something isn't right with the dataflow.
Sorry but why do you say that 20MB/s is not even close to a 1GbE link ? 1GbE link max speed is not 125MB/S ?

Thanks for your help tsightler
Gostev
Chief Product Officer
Posts: 31816
Liked: 7302 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Backup slow with EMC DD2500 and DDBoost

Post by Gostev »

Because 20 MB/s is not even close to 125 MB/s? :D
chrisdearden
Veteran
Posts: 1531
Liked: 226 times
Joined: Jul 21, 2010 9:47 am
Full Name: Chris Dearden
Contact:

Re: Backup slow with EMC DD2500 and DDBoost

Post by chrisdearden »

when you added the DD - which server did you run the boost gateway on?
Guyzmo
Expert
Posts: 140
Liked: 12 times
Joined: Jun 04, 2013 8:58 am
Full Name: Samir Bellili
Contact:

Re: Backup slow with EMC DD2500 and DDBoost

Post by Guyzmo » 1 person likes this post

You need to disable compression and inline dedup. (Best practice for DD repository)

Maybe this settings "enable" can reduce the speed.
Samir Bellili
Solution Architect France
tsightler
VP, Product Management
Posts: 6035
Liked: 2860 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Backup slow with EMC DD2500 and DDBoost

Post by tsightler »

Gostev wrote:Because 20 MB/s is not even close to 125 MB/s? :D
Exactly. The problem here is obviously not bandwidth, at least not directly. It may be dataflow not optimally using the data paths that you have, but with 1GbE you should be able to get far higher than even your previous speeds. Honestly, your original speeds look more like 100Mb speeds that 1Gb.

You mention that you are using a physical backup server, but I see at least two proxies in use, and one is using NBD mode (the screen was cut off so I couldn't see the other). Where is "VeeamProxy" located in relation to "VMware Backup Proxy"? And as previously asked, which system did you select as the gateway server to to DD? It looks like this new job is for some reason using VeeamProxy via NBD and that this server might be across a slow link. If it's also being used as the DD gateway then this is the problem and also explain why network is the bottleneck.

As a temporary troubleshooting step I'd suggest manually selecting the proxy for that job to use only the "VMware Backup Proxy" and see if that helps.
meilicke
Influencer
Posts: 22
Liked: 4 times
Joined: Sep 02, 2014 2:51 pm
Full Name: Scott Meilicke
Contact:

Re: Backup slow with EMC DD2500 and DDBoost

Post by meilicke » 2 people like this post

Just as a data point, we have:

* Veeam v8 Patch 1
* DD2500 head plus tray used as a boost repository. All jobs go here.
* 110 VMs, 12.8TB over four jobs.
* Between 115MB/s and 223MB/s processing rate last night.
* The total time required is about an 1 hour and a half. We are using boost repositories.
* The DD reports a throughput of around
* The source is our bottleneck.
* The DD is connected via two 1G links, not 10G.
rereduck
Enthusiast
Posts: 28
Liked: 1 time
Joined: Dec 09, 2013 9:11 am
Contact:

Re: Backup slow with EMC DD2500 and DDBoost

Post by rereduck »

Gostev wrote:Because 20 MB/s is not even close to 125 MB/s?
Oh ok ^^ , sorry I did not understand previous post from tsightler :)
chrisdearden wrote:when you added the DD - which server did you run the boost gateway on?
Automatic Selection
tsightler wrote: Exactly. The problem here is obviously not bandwidth, at least not directly. It may be dataflow not optimally using the data paths that you have, but with 1GbE you should be able to get far higher than even your previous speeds. Honestly, your original speeds look more like 100Mb speeds that 1Gb.

You mention that you are using a physical backup server, but I see at least two proxies in use, and one is using NBD mode (the screen was cut off so I couldn't see the other). Where is "VeeamProxy" located in relation to "VMware Backup Proxy"? And as previously asked, which system did you select as the gateway server to to DD? It looks like this new job is for some reason using VeeamProxy via NBD and that this server might be across a slow link. If it's also being used as the DD gateway then this is the problem and also explain why network is the bottleneck.

As a temporary troubleshooting step I'd suggest manually selecting the proxy for that job to use only the "VMware Backup Proxy" and see if that helps.
At first we only use one physical server as proxy but trying to improve performance I have add a second virtual proxy server to see if it is better with no results so I think I'll turn off virtual proxy.
Guyzmo wrote:You need to disable compression and inline dedup. (Best practice for DD repository)
Maybe this settings "enable" can reduce the speed.
Thanks Guyzmo, I think you have solved my problem :) With one job, speed is now arround 60MB/s and I backup 10,4TB in 8 hours instead of 20 hours ! I have followed recommandation in this KB : http://www.veeam.com/kb1745 (Why I didn't see it before ? ^^) !

I think I'll create 4 jobs instead of 1 like Meilicke to improve again performances. Thank to you for your feedback Meilicke :)

If you have any other advises to improve again performances, I'm interested :)
Guyzmo
Expert
Posts: 140
Liked: 12 times
Joined: Jun 04, 2013 8:58 am
Full Name: Samir Bellili
Contact:

Re: Backup slow with EMC DD2500 and DDBoost

Post by Guyzmo »

Hi

If you have followed the kb you have configure compression with dedupe friendly.

With ddboost, I think the best configuration for compression is None.
Can you test it and give us the speed?

Thanks.
Samir Bellili
Solution Architect France
SGalbincea
Enthusiast
Posts: 55
Liked: 6 times
Joined: May 25, 2012 2:09 pm
Full Name: Steve Galbincea
Location: Houston, TX
Contact:

Re: Backup slow with EMC DD2500 and DDBoost

Post by SGalbincea »

I am interested to know which works best as well. Currently we use dedupe friendly and get decent rates, but I may switch to none to see what happens.
Senior Solutions Engineer, SLED - VMware
Gostev
Chief Product Officer
Posts: 31816
Liked: 7302 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Backup slow with EMC DD2500 and DDBoost

Post by Gostev »

rereduck wrote:I have followed recommandation in this KB : http://www.veeam.com/kb1745
You may not have noticed, but this KB applies to versions 6.x, 7.x only.
B&R v8 and Data Domain users should refer to KB1956 instead.
emachabert
Veeam Vanguard
Posts: 395
Liked: 169 times
Joined: Nov 17, 2010 11:42 am
Full Name: Eric Machabert
Location: France
Contact:

Re: Backup slow with EMC DD2500 and DDBoost

Post by emachabert »

Is everything ok regarding network cabling ? Duplex and speed ? At both ends.
Regarding your setup (proxy using 10gb/s and DD using gbe), I would suspect pause frames (flow control) causing the slow down on the network path.
Is the proxy network graph looking like a crenel ?
Did you do a network capture to look for pause frame or massive TCP retransmission ?
Veeamizing your IT since 2009/ Veeam Vanguard 2015 - 2023
rereduck
Enthusiast
Posts: 28
Liked: 1 time
Joined: Dec 09, 2013 9:11 am
Contact:

Re: Backup slow with EMC DD2500 and DDBoost

Post by rereduck »

Guyzmo wrote:Hi

If you have followed the kb you have configure compression with dedupe friendly.

With ddboost, I think the best configuration for compression is None.
Can you test it and give us the speed?

Thanks.
I'll do it next week and I'll give you feedback about that.
Gostev wrote: You may not have noticed, but this KB applies to versions 6.x, 7.x only.
B&R v8 and Data Domain users should refer to KB1956 instead.
Thanks Gostev but in this KB is not about job settings and it says :

"For more information regarding Backup Job settings please reference:
http://www.veeam.com/kb1745"

Do you have any best practices for job settings and DataDomain update to version 8 ?

Thanks.
rereduck
Enthusiast
Posts: 28
Liked: 1 time
Joined: Dec 09, 2013 9:11 am
Contact:

Re: Backup slow with EMC DD2500 and DDBoost

Post by rereduck »

emachabert wrote:Is everything ok regarding network cabling ? Duplex and speed ? At both ends.
Regarding your setup (proxy using 10gb/s and DD using gbe), I would suspect pause frames (flow control) causing the slow down on the network path.
Is the proxy network graph looking like a crenel ?
Did you do a network capture to look for pause frame or massive TCP retransmission ?
Since I've set "compression mode " to "dedup friendly" and disable inline dedup in job settings, performance are much better and network is not a problem.

I'm always looking to reduce my backup time but I think I'll create 4 jobs instead of one to do that.

Thanks for your help.
Gostev
Chief Product Officer
Posts: 31816
Liked: 7302 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Backup slow with EMC DD2500 and DDBoost

Post by Gostev »

rereduck wrote:Do you have any best practices for job settings and DataDomain update to version 8?
v8 sets the recommended job settings automatically depending on the backup repository you chose for the job, so you no longer have to worry about that part.
SGalbincea
Enthusiast
Posts: 55
Liked: 6 times
Joined: May 25, 2012 2:09 pm
Full Name: Steve Galbincea
Location: Houston, TX
Contact:

Re: Backup slow with EMC DD2500 and DDBoost

Post by SGalbincea »

That sort of happens. When I click next it prompts to set the advanced settings, and it does indeed uncheck inline dedupe...but compression is set to optimal, not dedupe friendly or none which I understand are the best settings for DataDomain with DDBoost.
Senior Solutions Engineer, SLED - VMware
Guyzmo
Expert
Posts: 140
Liked: 12 times
Joined: Jun 04, 2013 8:58 am
Full Name: Samir Bellili
Contact:

Re: Backup slow with EMC DD2500 and DDBoost

Post by Guyzmo »

I think None is the best setting for speed but Dedupe Friendly is ok.

Could you test with Compression to None and compare speeds?
Samir Bellili
Solution Architect France
Gostev
Chief Product Officer
Posts: 31816
Liked: 7302 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Backup slow with EMC DD2500 and DDBoost

Post by Gostev » 1 person likes this post

SGalbincea wrote:compression is set to optimal, not dedupe friendly or none which I understand are the best settings for DataDomain with DDBoost
That is the correct statement.

Data Domain requires NO compression, which is why that advanced backup repository option to uncompress before saving is checked by default for DDBoost repositories. Whereas having Optimal compression enabled in the job reduces network bandwidth usage without any side effects.
Guyzmo wrote:Could you test with Compression to None and compare speeds?
Does not matter, as EMC requires that any sort of compression is disabled.
rereduck
Enthusiast
Posts: 28
Liked: 1 time
Joined: Dec 09, 2013 9:11 am
Contact:

Re: Backup slow with EMC DD2500 and DDBoost

Post by rereduck »

Hello, me again :)

I see no big difference between compression set to "none" and "dedup friendly".

I'm a trying to do 3 jobs instead of 1 to speed up backup time but since I've do that Veeam Backup server did not respond and freeze and backup failed.

I can see this error on Veeam Backup :

Failed to wait mutex ... :

Image

On my jobs I can see this error on all my VMs :

12/01/2015 20:01:22 :: Processing Squid Error: Microsoft SQL server hosting the configuration database is currently unavailable. Possible reasons are heavy load, networking issue, server reboot, or hot backup.
Please wait, and try again later.

Do you have any ideas why my server is freezing ?

My SQL Veeam Database is a SQL Server Express 2005 edition, can it be a problem ?

Here my infrastructure :

Veeam Backup Server and Proxy : 1 Physical Server with 2x10Gbe (SQL Express 2005, 12 core (24 logical) and 16G ram) in LAN Mode (no Direct SAN)
Veeam Proxy : 1 VM (8 core and 8G ram)
Repository : 1 DataDomain DD2500 connected with 2x10Gbe

Vmware :
80 VM and 10TB

Thanks
Post Reply

Who is online

Users browsing this forum: AdsBot [Google], Bing [Bot], damian321, Majestic-12 [Bot] and 54 guests