backup speed is so slow via DDboost

[baber]

I am using emc data domain as backup repository and i passed through and hba card on my veeam server (veeam backup & replication is a vm) and using DDboost my veeam server has 16vcpu thus i was set maximum concurrent task = 10 on backup proxy (backup proxy is my veeam server ) also maximum concurrent task on repository that is data domain is 10 now maximum bandwidth that i can use due to backup is 250MB/s why ? how can increase this speed ?

[Gostev]

Increasing concurrent tasks should help. Deduplicating storage appliances are special in the way that they have very limited throughput per stream, but "love" multiple streams. If I remember correctly, for higher-end Data Domain appliances those should be set to hundreds. I believe you should be able to find the recommended value of concurrent streams for your appliance model in the Data Domain documentation.

[baber]

Thanks but if i set concurent task on my repository that is data domain on 100 thus had to set 100 for concurrent task on my backup proxy
is that correct ?

[foggy]

Proxy and repository components are independent in terms of the number of concurrent tasks you assign to them (even though in your case both are running on a single server).

[baber]

I have 16vcpu on my veeam backup server that is a vm . now want to know
what is the best practice for me concurent task on backup proxy and backup repository (my backup repository is data domain )

BR

[neilp]

test the restore speed.... its really slow too for Data Domain.

[Gostev]

what is the best practice for me concurent task on backup proxy and backup repository (my backup repository is data domain)

The best practice to achieve good overall throughput is to make sure you send as many tasks to Data Domain as it can accept.

[baber]

thanks , but this is the main question how can understand i am sending maximum concurrent task to data domain ?
now i changed number of vcpu on veeam server and it has 24 cpu thus is set 24 cpu on backup proxy and set 220 concurrent task on data domain as backup repository

Is that ok ? or i can do better

[foggy]

You can act based on recommended settings outlined in Data Domain documentation and increase the number of tasks gradually, monitoring backup performance. We don't even know your DD model.

[baber]

my data domain model is 6800

[foggy]

Google says DD6800 supports up to 400 concurrent backup streams.

[Gostev]

now i changed number of vcpu on veeam server and it has 24 cpu thus is set 24 cpu on backup proxy and set 220 concurrent task on data domain as backup repository

This means Data Domain will never see more that 24 concurrent tasks landing on it, because 24 is the maximum amount of concurrent tasks your backup proxy can be sending. The fact that you allowed Data Domain to accept 220 concurrent tasks does not matter... it's like staffing 220 cash registers when you have no more 24 concurrent customers in your entire store at any given time.

[janezk]

I think, if you want to increase the number of concurrent tasks, you should add another proxy ( or more). Make sure it's VM is sitting on different HW than your first proxy, so that you distribute your load on your HW resources. ( obviously depending on the environment...)
Also check the settings on DataDomain configuration, I believe there could also be some limitations set.
And keep in mind that restore is very slow from DD.

BR Janez

[AdamWay]

We have a DD6300, 6 vm proxies configured for hotadd with 6 vCPU each, 10GB Ethernet whole way through for local backup. It chews through anything we throw at it
Changing from NBD to HotAdd increased our throughput considerably.

Also check that the IP/DNS you have added the DD repo as is configured for the Data NICS and not the management NIC of the DataDomain, made that mistake on day1 as we were too excited about getting it up and running.

[adb98]

There are a bunch of gotchas that can cause slow performance. A big start would be look at your jobs to see where the bottleneck is. It will tell you either source, target, network, etc. That is normally the direction to look at.

1st off... You should never ever use NBD for your transport unless you have a small shop and you are not using HA in VMware. NBD transport always uses the management port/nic on the host to pass traffic. If this gets saturated and HA is turned on, it can and will cause a host to fail over. As AdamWay stated the easiest way is to use HotAdd. You still have to be careful because now your using the VM traffic nics to pass the traffic but at least its not over the management nic. If you don't have a 10gb nic and a lot of VMs on a host I would advise against it as well as VMs could start loosing traffic.

The first thing we need to know is how are you using DDBoost.... Are you going over the SAN or are you going over cooper? Next would be what speed. I would recommend 10gb copper if you have it. Use single nics and setup an interface group in DDBoost. This will balance out the nics when backing up and restoring. Ensure that each IP you use in the interface group is accesable to the main veeam server and transports.

Also check that you have active connections when looking inside DDBoost on the DD. You have to setup allowed clients. Putting a wildcard will allow all but is less secure. If you don't have connections...DDBoost is not working and more than likely not configured correctly.

Also on the DD...How many Mtrees do you have setup and using for this. Streams are based upon Mtrees. So if you only have one Mtree.... don't. You should have multiple Mtrees setup to spread out the streams. I think we have at least 20 right now. Also streams get taken up when using DDOS replication to another DD or the cloud so keep that in mind.

I could go on and on about other gotchas but the last one I will leave you with is the version DDOS. Make sure its current and supported by Veeam. This can cause issues.

[Ci2Group]

Veeam support 06149322
We are seeing poor performance with the our DataDomain implementation. I am curious about the comment about having only 1 mtree. In the documentation that I have found there is no mention of how to configure or how Mtrees can affect performance. It does not seem like anyone in support knows , I just keep getting pointed at the same docs. Can anyone elaborate on Mtree config ?
On the data domain we are only seeing 4 write streams at peak times and this is not giving us the performance we would expect or what we need to meet RTO. Veeam jobs are showing the 'target' as the primary bottleneck. Our gateway server is a monster physical box with 64 cores and 768GB RAM and 10GB network. The datadomain has all interfaces grouped with total bandwidth of 40GB. During backups, network bandwidth does not exceed 5000Mbit.

[janezk]

The comment of only one mtree also interests me. We have physical proxy servers with DDboost via SAN connection to DD. We have set up a scale-out BR with one mtree per proxy. Would it be better to have multiple mtrees per server (in SOBR)? Have been searching for this info for a long time, but never got the answer!

[Ci2Group]

We have implemented an mtree per proxy and each proxy is also a gateway server. This has increased our backup speeds significantly. There are some articles that suggest creating a folder in an mtree has the same effect. The important thing is that you are using a SOBR and there are multiple gateway servers to send data to DataDomain.