-
- Enthusiast
- Posts: 28
- Liked: 1 time
- Joined: Dec 09, 2013 9:11 am
- Contact:
Backup slow with EMC DD2500 and DDBoost
Hello,
Case ID : 00720532
We have just acquire an EMC DD2500 with DDBoost and Veeam V8 (with Patch 1 since yesterday) and backups are very slow.
Before DD2500 we backup our VM with Veeam V7 and V8 on local disk on a physical server with only one proxy (the physical server).
With this configuration, we managed to daily backup 10TB in incremental mode for 70/80 vm in 3h to 6h.Bottleneck was Source (99%/16%/2%/0%) and processing rate was approximativly 60MB/s.
With DD2500 and DDBoost, we need between 15 and 28 hours to backup 10TB in incremental mode.Bottleneck is Network (50%/11%/64%/41%) and processing rate is approximativly 20MB/s.
Do you think is it normal ? Is it a configuration issue ? How can I reduce the backup Window ?
To sump up :
Before :
One Physcial Server with 30Tb disk
One Proxy (the psysical server)
2x 10GB/s for the network.
One Veeam Job with 70VM and 10TB in daily incremental and Synthetic full once a week.
Compression leve : High
Storage Optimization : Local Target
Inlinde dedup : Enabled
Time to backup : between 3h and 6h
Now :
DD2500 with 45TB
4x 1GB/s for the network (We plan to migrate to 2x10Gb/s)
Two Proxy (The Physical server and one VM)
One Veeam Job with 70VM and 10TB in daily incremental and Synthetic full once a week.
Compression leve : Optimal
Storage Optimization : LAN Target
Inlinde dedup : Enabled
Time to backup : between 15h and 28h.
Virtual infrastructure :
Vmware Vsphere 5.5
Thanks for your help
Case ID : 00720532
We have just acquire an EMC DD2500 with DDBoost and Veeam V8 (with Patch 1 since yesterday) and backups are very slow.
Before DD2500 we backup our VM with Veeam V7 and V8 on local disk on a physical server with only one proxy (the physical server).
With this configuration, we managed to daily backup 10TB in incremental mode for 70/80 vm in 3h to 6h.Bottleneck was Source (99%/16%/2%/0%) and processing rate was approximativly 60MB/s.
With DD2500 and DDBoost, we need between 15 and 28 hours to backup 10TB in incremental mode.Bottleneck is Network (50%/11%/64%/41%) and processing rate is approximativly 20MB/s.
Do you think is it normal ? Is it a configuration issue ? How can I reduce the backup Window ?
To sump up :
Before :
One Physcial Server with 30Tb disk
One Proxy (the psysical server)
2x 10GB/s for the network.
One Veeam Job with 70VM and 10TB in daily incremental and Synthetic full once a week.
Compression leve : High
Storage Optimization : Local Target
Inlinde dedup : Enabled
Time to backup : between 3h and 6h
Now :
DD2500 with 45TB
4x 1GB/s for the network (We plan to migrate to 2x10Gb/s)
Two Proxy (The Physical server and one VM)
One Veeam Job with 70VM and 10TB in daily incremental and Synthetic full once a week.
Compression leve : Optimal
Storage Optimization : LAN Target
Inlinde dedup : Enabled
Time to backup : between 15h and 28h.
Virtual infrastructure :
Vmware Vsphere 5.5
Thanks for your help
-
- Chief Product Officer
- Posts: 31816
- Liked: 7302 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: Backup slow with EMC DD2500 and DDBoost
I would have support start from investigating "network" being a bottleneck on 4Gb Ethernet with mere 20MB/s processing rate, as this looks extremely suspicious. Your current bottleneck stats basically mean that the job spends most time waiting for the data to be transferred over network... backup target type does not even matter in such conditions. Thanks!
-
- Enthusiast
- Posts: 28
- Liked: 1 time
- Joined: Dec 09, 2013 9:11 am
- Contact:
Re: Backup slow with EMC DD2500 and DDBoost
Thanks Gostev for your answer.
I've read that Veeam use only one network link to transfer data (because there only one Job) so only 1Gb in my case that's why we have planned to migrate from 4x 1Gb to 2x10Gb but I was not sure thaht it was a network issue.
So I think I'll wait for 2x10Gb to see if it is better.
Thanks
I've read that Veeam use only one network link to transfer data (because there only one Job) so only 1Gb in my case that's why we have planned to migrate from 4x 1Gb to 2x10Gb but I was not sure thaht it was a network issue.
So I think I'll wait for 2x10Gb to see if it is better.
Thanks
-
- Chief Product Officer
- Posts: 31816
- Liked: 7302 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: Backup slow with EMC DD2500 and DDBoost
Not if you actually teamed the network links in Windows, in which case it is OS itself that will leverage all 4 links for bandwidth aggregation. You are right that Veeam by itself cannot team multiple links, but we don't implement this simply because teaming is normally done on lower OSI stack level.
-
- VP, Product Management
- Posts: 6035
- Liked: 2860 times
- Joined: Jun 05, 2009 12:57 pm
- Full Name: Tom Sightler
- Contact:
Re: Backup slow with EMC DD2500 and DDBoost
And Veeam does use mulitple TCP connections, so assuming the load balancing used by the team is based on TCP port then you should be able to leverage mulitple links. That being said, you're talking about speeds of 20MB/s here, that's not even close to a single 1GbE link so clearly something isn't right with the dataflow.
-
- Enthusiast
- Posts: 28
- Liked: 1 time
- Joined: Dec 09, 2013 9:11 am
- Contact:
Re: Backup slow with EMC DD2500 and DDBoost
To complete my first post, here are screenshot of the same virtual machine with DD2500 and with our Physical Server and Local Disks :
-
- Enthusiast
- Posts: 28
- Liked: 1 time
- Joined: Dec 09, 2013 9:11 am
- Contact:
Re: Backup slow with EMC DD2500 and DDBoost
Sorry but why do you say that 20MB/s is not even close to a 1GbE link ? 1GbE link max speed is not 125MB/S ?tsightler wrote:And Veeam does use mulitple TCP connections, so assuming the load balancing used by the team is based on TCP port then you should be able to leverage mulitple links. That being said, you're talking about speeds of 20MB/s here, that's not even close to a single 1GbE link so clearly something isn't right with the dataflow.
Thanks for your help tsightler
-
- Chief Product Officer
- Posts: 31816
- Liked: 7302 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: Backup slow with EMC DD2500 and DDBoost
Because 20 MB/s is not even close to 125 MB/s?
-
- Veteran
- Posts: 1531
- Liked: 226 times
- Joined: Jul 21, 2010 9:47 am
- Full Name: Chris Dearden
- Contact:
Re: Backup slow with EMC DD2500 and DDBoost
when you added the DD - which server did you run the boost gateway on?
-
- Expert
- Posts: 140
- Liked: 12 times
- Joined: Jun 04, 2013 8:58 am
- Full Name: Samir Bellili
- Contact:
Re: Backup slow with EMC DD2500 and DDBoost
You need to disable compression and inline dedup. (Best practice for DD repository)
Maybe this settings "enable" can reduce the speed.
Maybe this settings "enable" can reduce the speed.
Samir Bellili
Solution Architect France
Solution Architect France
-
- VP, Product Management
- Posts: 6035
- Liked: 2860 times
- Joined: Jun 05, 2009 12:57 pm
- Full Name: Tom Sightler
- Contact:
Re: Backup slow with EMC DD2500 and DDBoost
Exactly. The problem here is obviously not bandwidth, at least not directly. It may be dataflow not optimally using the data paths that you have, but with 1GbE you should be able to get far higher than even your previous speeds. Honestly, your original speeds look more like 100Mb speeds that 1Gb.Gostev wrote:Because 20 MB/s is not even close to 125 MB/s?
You mention that you are using a physical backup server, but I see at least two proxies in use, and one is using NBD mode (the screen was cut off so I couldn't see the other). Where is "VeeamProxy" located in relation to "VMware Backup Proxy"? And as previously asked, which system did you select as the gateway server to to DD? It looks like this new job is for some reason using VeeamProxy via NBD and that this server might be across a slow link. If it's also being used as the DD gateway then this is the problem and also explain why network is the bottleneck.
As a temporary troubleshooting step I'd suggest manually selecting the proxy for that job to use only the "VMware Backup Proxy" and see if that helps.
-
- Influencer
- Posts: 22
- Liked: 4 times
- Joined: Sep 02, 2014 2:51 pm
- Full Name: Scott Meilicke
- Contact:
Re: Backup slow with EMC DD2500 and DDBoost
Just as a data point, we have:
* Veeam v8 Patch 1
* DD2500 head plus tray used as a boost repository. All jobs go here.
* 110 VMs, 12.8TB over four jobs.
* Between 115MB/s and 223MB/s processing rate last night.
* The total time required is about an 1 hour and a half. We are using boost repositories.
* The DD reports a throughput of around
* The source is our bottleneck.
* The DD is connected via two 1G links, not 10G.
* Veeam v8 Patch 1
* DD2500 head plus tray used as a boost repository. All jobs go here.
* 110 VMs, 12.8TB over four jobs.
* Between 115MB/s and 223MB/s processing rate last night.
* The total time required is about an 1 hour and a half. We are using boost repositories.
* The DD reports a throughput of around
* The source is our bottleneck.
* The DD is connected via two 1G links, not 10G.
-
- Enthusiast
- Posts: 28
- Liked: 1 time
- Joined: Dec 09, 2013 9:11 am
- Contact:
Re: Backup slow with EMC DD2500 and DDBoost
Oh ok ^^ , sorry I did not understand previous post from tsightlerGostev wrote:Because 20 MB/s is not even close to 125 MB/s?
Automatic Selectionchrisdearden wrote:when you added the DD - which server did you run the boost gateway on?
At first we only use one physical server as proxy but trying to improve performance I have add a second virtual proxy server to see if it is better with no results so I think I'll turn off virtual proxy.tsightler wrote: Exactly. The problem here is obviously not bandwidth, at least not directly. It may be dataflow not optimally using the data paths that you have, but with 1GbE you should be able to get far higher than even your previous speeds. Honestly, your original speeds look more like 100Mb speeds that 1Gb.
You mention that you are using a physical backup server, but I see at least two proxies in use, and one is using NBD mode (the screen was cut off so I couldn't see the other). Where is "VeeamProxy" located in relation to "VMware Backup Proxy"? And as previously asked, which system did you select as the gateway server to to DD? It looks like this new job is for some reason using VeeamProxy via NBD and that this server might be across a slow link. If it's also being used as the DD gateway then this is the problem and also explain why network is the bottleneck.
As a temporary troubleshooting step I'd suggest manually selecting the proxy for that job to use only the "VMware Backup Proxy" and see if that helps.
Thanks Guyzmo, I think you have solved my problem With one job, speed is now arround 60MB/s and I backup 10,4TB in 8 hours instead of 20 hours ! I have followed recommandation in this KB : http://www.veeam.com/kb1745 (Why I didn't see it before ? ^^) !Guyzmo wrote:You need to disable compression and inline dedup. (Best practice for DD repository)
Maybe this settings "enable" can reduce the speed.
I think I'll create 4 jobs instead of 1 like Meilicke to improve again performances. Thank to you for your feedback Meilicke
If you have any other advises to improve again performances, I'm interested
-
- Expert
- Posts: 140
- Liked: 12 times
- Joined: Jun 04, 2013 8:58 am
- Full Name: Samir Bellili
- Contact:
Re: Backup slow with EMC DD2500 and DDBoost
Hi
If you have followed the kb you have configure compression with dedupe friendly.
With ddboost, I think the best configuration for compression is None.
Can you test it and give us the speed?
Thanks.
If you have followed the kb you have configure compression with dedupe friendly.
With ddboost, I think the best configuration for compression is None.
Can you test it and give us the speed?
Thanks.
Samir Bellili
Solution Architect France
Solution Architect France
-
- Enthusiast
- Posts: 55
- Liked: 6 times
- Joined: May 25, 2012 2:09 pm
- Full Name: Steve Galbincea
- Location: Houston, TX
- Contact:
Re: Backup slow with EMC DD2500 and DDBoost
I am interested to know which works best as well. Currently we use dedupe friendly and get decent rates, but I may switch to none to see what happens.
Senior Solutions Engineer, SLED - VMware
-
- Chief Product Officer
- Posts: 31816
- Liked: 7302 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: Backup slow with EMC DD2500 and DDBoost
You may not have noticed, but this KB applies to versions 6.x, 7.x only.rereduck wrote:I have followed recommandation in this KB : http://www.veeam.com/kb1745
B&R v8 and Data Domain users should refer to KB1956 instead.
-
- Veeam Vanguard
- Posts: 395
- Liked: 169 times
- Joined: Nov 17, 2010 11:42 am
- Full Name: Eric Machabert
- Location: France
- Contact:
Re: Backup slow with EMC DD2500 and DDBoost
Is everything ok regarding network cabling ? Duplex and speed ? At both ends.
Regarding your setup (proxy using 10gb/s and DD using gbe), I would suspect pause frames (flow control) causing the slow down on the network path.
Is the proxy network graph looking like a crenel ?
Did you do a network capture to look for pause frame or massive TCP retransmission ?
Regarding your setup (proxy using 10gb/s and DD using gbe), I would suspect pause frames (flow control) causing the slow down on the network path.
Is the proxy network graph looking like a crenel ?
Did you do a network capture to look for pause frame or massive TCP retransmission ?
Veeamizing your IT since 2009/ Veeam Vanguard 2015 - 2023
-
- Enthusiast
- Posts: 28
- Liked: 1 time
- Joined: Dec 09, 2013 9:11 am
- Contact:
Re: Backup slow with EMC DD2500 and DDBoost
I'll do it next week and I'll give you feedback about that.Guyzmo wrote:Hi
If you have followed the kb you have configure compression with dedupe friendly.
With ddboost, I think the best configuration for compression is None.
Can you test it and give us the speed?
Thanks.
Thanks Gostev but in this KB is not about job settings and it says :Gostev wrote: You may not have noticed, but this KB applies to versions 6.x, 7.x only.
B&R v8 and Data Domain users should refer to KB1956 instead.
"For more information regarding Backup Job settings please reference:
http://www.veeam.com/kb1745"
Do you have any best practices for job settings and DataDomain update to version 8 ?
Thanks.
-
- Enthusiast
- Posts: 28
- Liked: 1 time
- Joined: Dec 09, 2013 9:11 am
- Contact:
Re: Backup slow with EMC DD2500 and DDBoost
Since I've set "compression mode " to "dedup friendly" and disable inline dedup in job settings, performance are much better and network is not a problem.emachabert wrote:Is everything ok regarding network cabling ? Duplex and speed ? At both ends.
Regarding your setup (proxy using 10gb/s and DD using gbe), I would suspect pause frames (flow control) causing the slow down on the network path.
Is the proxy network graph looking like a crenel ?
Did you do a network capture to look for pause frame or massive TCP retransmission ?
I'm always looking to reduce my backup time but I think I'll create 4 jobs instead of one to do that.
Thanks for your help.
-
- Chief Product Officer
- Posts: 31816
- Liked: 7302 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: Backup slow with EMC DD2500 and DDBoost
v8 sets the recommended job settings automatically depending on the backup repository you chose for the job, so you no longer have to worry about that part.rereduck wrote:Do you have any best practices for job settings and DataDomain update to version 8?
-
- Enthusiast
- Posts: 55
- Liked: 6 times
- Joined: May 25, 2012 2:09 pm
- Full Name: Steve Galbincea
- Location: Houston, TX
- Contact:
Re: Backup slow with EMC DD2500 and DDBoost
That sort of happens. When I click next it prompts to set the advanced settings, and it does indeed uncheck inline dedupe...but compression is set to optimal, not dedupe friendly or none which I understand are the best settings for DataDomain with DDBoost.
Senior Solutions Engineer, SLED - VMware
-
- Expert
- Posts: 140
- Liked: 12 times
- Joined: Jun 04, 2013 8:58 am
- Full Name: Samir Bellili
- Contact:
Re: Backup slow with EMC DD2500 and DDBoost
I think None is the best setting for speed but Dedupe Friendly is ok.
Could you test with Compression to None and compare speeds?
Could you test with Compression to None and compare speeds?
Samir Bellili
Solution Architect France
Solution Architect France
-
- Chief Product Officer
- Posts: 31816
- Liked: 7302 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: Backup slow with EMC DD2500 and DDBoost
That is the correct statement.SGalbincea wrote:compression is set to optimal, not dedupe friendly or none which I understand are the best settings for DataDomain with DDBoost
Data Domain requires NO compression, which is why that advanced backup repository option to uncompress before saving is checked by default for DDBoost repositories. Whereas having Optimal compression enabled in the job reduces network bandwidth usage without any side effects.
Does not matter, as EMC requires that any sort of compression is disabled.Guyzmo wrote:Could you test with Compression to None and compare speeds?
-
- Enthusiast
- Posts: 28
- Liked: 1 time
- Joined: Dec 09, 2013 9:11 am
- Contact:
Re: Backup slow with EMC DD2500 and DDBoost
Hello, me again
I see no big difference between compression set to "none" and "dedup friendly".
I'm a trying to do 3 jobs instead of 1 to speed up backup time but since I've do that Veeam Backup server did not respond and freeze and backup failed.
I can see this error on Veeam Backup :
Failed to wait mutex ... :
On my jobs I can see this error on all my VMs :
12/01/2015 20:01:22 :: Processing Squid Error: Microsoft SQL server hosting the configuration database is currently unavailable. Possible reasons are heavy load, networking issue, server reboot, or hot backup.
Please wait, and try again later.
Do you have any ideas why my server is freezing ?
My SQL Veeam Database is a SQL Server Express 2005 edition, can it be a problem ?
Here my infrastructure :
Veeam Backup Server and Proxy : 1 Physical Server with 2x10Gbe (SQL Express 2005, 12 core (24 logical) and 16G ram) in LAN Mode (no Direct SAN)
Veeam Proxy : 1 VM (8 core and 8G ram)
Repository : 1 DataDomain DD2500 connected with 2x10Gbe
Vmware :
80 VM and 10TB
Thanks
I see no big difference between compression set to "none" and "dedup friendly".
I'm a trying to do 3 jobs instead of 1 to speed up backup time but since I've do that Veeam Backup server did not respond and freeze and backup failed.
I can see this error on Veeam Backup :
Failed to wait mutex ... :
On my jobs I can see this error on all my VMs :
12/01/2015 20:01:22 :: Processing Squid Error: Microsoft SQL server hosting the configuration database is currently unavailable. Possible reasons are heavy load, networking issue, server reboot, or hot backup.
Please wait, and try again later.
Do you have any ideas why my server is freezing ?
My SQL Veeam Database is a SQL Server Express 2005 edition, can it be a problem ?
Here my infrastructure :
Veeam Backup Server and Proxy : 1 Physical Server with 2x10Gbe (SQL Express 2005, 12 core (24 logical) and 16G ram) in LAN Mode (no Direct SAN)
Veeam Proxy : 1 VM (8 core and 8G ram)
Repository : 1 DataDomain DD2500 connected with 2x10Gbe
Vmware :
80 VM and 10TB
Thanks
Who is online
Users browsing this forum: AdsBot [Google], Bing [Bot], damian321, Majestic-12 [Bot] and 54 guests