Comprehensive data protection for all workloads
Post Reply
baber
Expert
Posts: 660
Liked: 3 times
Joined: Nov 21, 2013 12:02 pm
Full Name: Babak Seyedi nejad
Contact:

backup speed is so slow via DDboost

Post by baber » Aug 07, 2019 1:06 pm

I am using emc data domain as backup repository and i passed through and hba card on my veeam server (veeam backup & replication is a vm) and using DDboost my veeam server has 16vcpu thus i was set maximum concurrent task = 10 on backup proxy (backup proxy is my veeam server ) also maximum concurrent task on repository that is data domain is 10 now maximum bandwidth that i can use due to backup is 250MB/s why ? how can increase this speed ?

Gostev
SVP, Product Management
Posts: 24784
Liked: 3513 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: backup speed is so slow via DDboost

Post by Gostev » Aug 07, 2019 3:03 pm

Increasing concurrent tasks should help. Deduplicating storage appliances are special in the way that they have very limited throughput per stream, but "love" multiple streams. If I remember correctly, for higher-end Data Domain appliances those should be set to hundreds. I believe you should be able to find the recommended value of concurrent streams for your appliance model in the Data Domain documentation.

baber
Expert
Posts: 660
Liked: 3 times
Joined: Nov 21, 2013 12:02 pm
Full Name: Babak Seyedi nejad
Contact:

Re: backup speed is so slow via DDboost

Post by baber » Aug 07, 2019 4:20 pm

Thanks but if i set concurent task on my repository that is data domain on 100 thus had to set 100 for concurrent task on my backup proxy
is that correct ?

foggy
Veeam Software
Posts: 18239
Liked: 1556 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: backup speed is so slow via DDboost

Post by foggy » Aug 07, 2019 4:30 pm

Proxy and repository components are independent in terms of the number of concurrent tasks you assign to them (even though in your case both are running on a single server).

baber
Expert
Posts: 660
Liked: 3 times
Joined: Nov 21, 2013 12:02 pm
Full Name: Babak Seyedi nejad
Contact:

Re: backup speed is so slow via DDboost

Post by baber » Aug 07, 2019 4:57 pm

I have 16vcpu on my veeam backup server that is a vm . now want to know
what is the best practice for me concurent task on backup proxy and backup repository (my backup repository is data domain )

BR

neilp
Lurker
Posts: 2
Liked: never
Joined: Jan 11, 2019 7:00 pm
Full Name: neil pedrosa
Contact:

Re: backup speed is so slow via DDboost

Post by neilp » Aug 08, 2019 2:03 am

test the restore speed.... its really slow too for Data Domain.

Gostev
SVP, Product Management
Posts: 24784
Liked: 3513 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: backup speed is so slow via DDboost

Post by Gostev » Aug 08, 2019 10:45 am

baber wrote:
Aug 07, 2019 4:57 pm
what is the best practice for me concurent task on backup proxy and backup repository (my backup repository is data domain)
The best practice to achieve good overall throughput is to make sure you send as many tasks to Data Domain as it can accept.

baber
Expert
Posts: 660
Liked: 3 times
Joined: Nov 21, 2013 12:02 pm
Full Name: Babak Seyedi nejad
Contact:

Re: backup speed is so slow via DDboost

Post by baber » Aug 08, 2019 11:57 am

thanks , but this is the main question how can understand i am sending maximum concurrent task to data domain ?
now i changed number of vcpu on veeam server and it has 24 cpu thus is set 24 cpu on backup proxy and set 220 concurrent task on data domain as backup repository

Is that ok ? or i can do better

foggy
Veeam Software
Posts: 18239
Liked: 1556 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: backup speed is so slow via DDboost

Post by foggy » Aug 08, 2019 12:22 pm

You can act based on recommended settings outlined in Data Domain documentation and increase the number of tasks gradually, monitoring backup performance. We don't even know your DD model.

baber
Expert
Posts: 660
Liked: 3 times
Joined: Nov 21, 2013 12:02 pm
Full Name: Babak Seyedi nejad
Contact:

Re: backup speed is so slow via DDboost

Post by baber » Aug 08, 2019 12:42 pm

my data domain model is 6800

foggy
Veeam Software
Posts: 18239
Liked: 1556 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: backup speed is so slow via DDboost

Post by foggy » Aug 08, 2019 1:46 pm 3 people like this post

Google says DD6800 supports up to 400 concurrent backup streams.

Gostev
SVP, Product Management
Posts: 24784
Liked: 3513 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: backup speed is so slow via DDboost

Post by Gostev » Aug 08, 2019 2:21 pm

baber wrote:
Aug 08, 2019 11:57 am
now i changed number of vcpu on veeam server and it has 24 cpu thus is set 24 cpu on backup proxy and set 220 concurrent task on data domain as backup repository
This means Data Domain will never see more that 24 concurrent tasks landing on it, because 24 is the maximum amount of concurrent tasks your backup proxy can be sending. The fact that you allowed Data Domain to accept 220 concurrent tasks does not matter... it's like staffing 220 cash registers when you have no more 24 concurrent customers in your entire store at any given time.

janezk
Enthusiast
Posts: 42
Liked: 5 times
Joined: Jul 25, 2016 10:42 am
Full Name: Janez K
Location: Slovenija
Contact:

Re: backup speed is so slow via DDboost

Post by janezk » Aug 12, 2019 5:41 am

I think, if you want to increase the number of concurrent tasks, you should add another proxy ( or more). Make sure it's VM is sitting on different HW than your first proxy, so that you distribute your load on your HW resources. ( obviously depending on the environment...)
Also check the settings on DataDomain configuration, I believe there could also be some limitations set.
And keep in mind that restore is very slow from DD.

BR Janez

AdamWay
Novice
Posts: 8
Liked: 5 times
Joined: Jul 03, 2013 1:32 pm
Full Name: Adam Way
Contact:

Re: backup speed is so slow via DDboost

Post by AdamWay » Aug 12, 2019 8:40 am 1 person likes this post

We have a DD6300, 6 vm proxies configured for hotadd with 6 vCPU each, 10GB Ethernet whole way through for local backup. It chews through anything we throw at it
Changing from NBD to HotAdd increased our throughput considerably.

Also check that the IP/DNS you have added the DD repo as is configured for the Data NICS and not the management NIC of the DataDomain, made that mistake on day1 as we were too excited about getting it up and running.

adb98
Enthusiast
Posts: 49
Liked: 8 times
Joined: Jul 21, 2016 5:03 pm
Full Name: Aaron B
Contact:

Re: backup speed is so slow via DDboost

Post by adb98 » Aug 12, 2019 1:45 pm

There are a bunch of gotchas that can cause slow performance. A big start would be look at your jobs to see where the bottleneck is. It will tell you either source, target, network, etc. That is normally the direction to look at.

1st off... You should never ever use NBD for your transport unless you have a small shop and you are not using HA in VMware. NBD transport always uses the management port/nic on the host to pass traffic. If this gets saturated and HA is turned on, it can and will cause a host to fail over. As AdamWay stated the easiest way is to use HotAdd. You still have to be careful because now your using the VM traffic nics to pass the traffic but at least its not over the management nic. If you don't have a 10gb nic and a lot of VMs on a host I would advise against it as well as VMs could start loosing traffic.

The first thing we need to know is how are you using DDBoost.... Are you going over the SAN or are you going over cooper? Next would be what speed. I would recommend 10gb copper if you have it. Use single nics and setup an interface group in DDBoost. This will balance out the nics when backing up and restoring. Ensure that each IP you use in the interface group is accesable to the main veeam server and transports.

Also check that you have active connections when looking inside DDBoost on the DD. You have to setup allowed clients. Putting a wildcard will allow all but is less secure. If you don't have connections...DDBoost is not working and more than likely not configured correctly.

Also on the DD...How many Mtrees do you have setup and using for this. Streams are based upon Mtrees. So if you only have one Mtree.... don't. You should have multiple Mtrees setup to spread out the streams. I think we have at least 20 right now. Also streams get taken up when using DDOS replication to another DD or the cloud so keep that in mind.

I could go on and on about other gotchas but the last one I will leave you with is the version DDOS. Make sure its current and supported by Veeam. This can cause issues.

Post Reply

Who is online

Users browsing this forum: Bing [Bot], Google [Bot], john_wood, Rick.Vanover, Shestakov and 15 guests