Veeam and DataDomain Backup Perfomance

[nbaptista]

Hi,

I've searched in this forum for answers about the issues I'm having but unfortunately I didn't found a solution. Meanwhile I've seen many posts talking about DataDomain, but I hope you don't mind to open another one, because the issue I have is basically weird.

In my scenario, I have Veeam solution installed with 4 dedicated VM's (8vCPU, 16GB Ram) working as Veeam Backup Proxy (transport as virtual appliance) and as Gateway Server to a DataDomain DD6400 appliance using DDBoost feature. The 4 VM's that are distributed where each one is running in each host/node of our VMware cluster, so we have 4 physical hosts.

Our network connectivity is 10Gb for VMware hosts, DataDomain appliance, everything using the same 10Gb switches.

Since we have installed the DataDomain solution, I've noticed that backing up multiple VM's, the data transfer to the appliance is high, peaking 2GB/s, and 1GB/s as average.
But in case I'm running backup jobs which have big VM's, like for example 1 backup job with only 1 VM but with 20TB, it takes a lot of time to complete/finish, basically I've one running at 100MB's for hours/days. Looking at the backup job task, it states that the bottleneck is at the target, but how?! Looking in the backup proxy/GW server which is being currently used by the backup job, its using about 20-30% CPU, with some network traffic around 500-800Mb/s.
Looking at the DataDomain performance charts, the disk usage is around 100MBs, very low CPU usage, and from what I know the DD appliance has some sort of a dedicated hardware accelerator card (Intel QuickAssist Adapter 8970) for data compression/encryption, so if there is any performances status about it, it's not something accessible via WebGUI

So for an appliance that even has an option for having 25Gb/s network connectivity, how can this appliance actually be able to achieve that such performance? By only running hundreds of VM backups in parallel?
And what about jobs with big VM's, we have to wait days to get them completely backed up?!

Btw, according to Dell product overview:
https://www.dell.com/en-us/blog/upgradi ... ct-dd6400/

They state that model have support throughput of up to 12.7 TB/hr and when using DD Boost can have throughput speed of up to 27.7 TB/hr, which in my case with DDBoost clearly is not even capable of doing 1TB/hr...

If someone had a similar issue, could you please share your experience and possible solution, if there's any?

Thanks

[david.domask]

Hi Nelson, welcome to the forums.

The issue isn't so much the network speed but rather that with Data Domain, it's necessary to understand that there is a limit on the throughput for an individual read/write stream. This topic explains a bit of theory-crafting around Data Domain write streams in further detail but in brief, the high throughput of Data Domains and other deduplication appliances tends to come from parallelization and total throughput, not the max speed of an individual write stream.

So when you're streaming many machines to the Data Domain, you will see improved processing rate -- likely you are maxing out each stream or getting pretty close, but because the write stream has a max throughput*, a single stream writing to the Data Domain will not saturate the network connection.

One thing I would suggest is just ensure that the gateway/proxy server is sized appropriately for the workload. Gateway role must be sized as per the User Guide, and this is in addition to the system requirements for being a backup proxy; I know you mentioned you weren't seeing resource contention, but I think under load you will probably see a difference. To be clear though, I'm doubtful that this will have a noticeable impact on the single-stream writes, but it is best to size the proxy/gateway appropriately anyways and seeing if the single-stream performance improves.

* I don't have hard numbers on this but I remember seeing an old chart floating around for DD9400 that maxed out around 310 MB/s for a single write stream under ideal conditions.

[foggy]

Hi Nelson,

You've nailed it here:

So for an appliance that even has an option for having 25Gb/s network connectivity, how can this appliance actually be able to achieve that such performance? By only running hundreds of VM backups in parallel?

Exactly. The ingestion rate of a single write stream on dedupe appliances is typically limited, so you need multi-VM/multi-disk/parallel jobs to achieve better throughput.

[nbaptista]

Hi @david.domask and @foggy

Thank you for sharing!

I totally understand/agree about having the pre-requisites according to Veeam documentation, and as soon my current single "big" VM backup finishes, I have on my list to test on building a VM dedicated only to be the GW server to DataDomain appliance to understand if it makes any difference.

About the appliance only being good using a "big list" of multiple backups at the same time/parallel, I'm sorry but it makes some confusion to me, because if while occurring multiple backups at the same time which for the appliance it would have to receive in a brief time range a big quantity of data in multi stream and it has to handle the dedup/compression with all that data, why it can't handle big quantity of data but using a single stream?

Btw, if someone has a DD6400 model, can you please put here the average in data transfers using a single stream for comparison?

Thanks

[foggy]

We have 6400 in our internal lab. We see 155 MB/s for a single write stream over Ethernet and 270 MB/s for read.

[gmajestix]

I put like 40 VMs in a single job an performed an active full backup to DD6400. At the start it was backing up with around 5GB/s. Later this speed dropped as there were only a few VMs still backing up. Proxy servers were Hyper-V hosts. However some context is needed here. The source storage was all flash storage and all the content was previously already written to DD6400. DDBoost did its job and network was utilized around 400Mbit to 500Mbit. Single stream stream restore was around 100MB/s.