New Data Domain . Veeam move backups really slow

[pmichelli]

We just got a new Data Domain to replace and older model. It can write backups at almost full 10GB/s speeds. I am trying to move the backups from the old DD to the new DD by changing the repository and then selecting move backups when Veeam prompts. I am getting at most 100MB/sec. Some VMs in the job are moving data at 5MB/sec. I have almost 130TB of data to move, at this rate it will take weeks and I won't be able to run backups while this is taking place.

Can anyone explain why this is so painfully slow? Does it have anything to do with the fact is has to hydrate the data, move it and then re dedupe it on the other end? Is there anything we can do to make this move faster?

[david.domask]

Hi pmichelli,

Any chance there's 1 Gbit NIC in play somewhere? Maybe one of the Datadomain management connections instead of the backup traffic connections? Similarly, as a Gateway Server is required for DataDomain DDBoost connections, is there perhaps 1 Gbit on the gateway server?

It's just that it seems to cap at 100 MB/s makes me think there's a 1 Gbit somewhere in the route.

I don't think this is about the rehydration -- are you seeing the same speeds on restore?

I recommend open a Support case and let Veeam Support check the logs if the above doesn't seem to match immediately or if you cannot spot the bottleneck; remember to include logs -- use the 3rd radio option when exporting logs and select the Veeam server itself to export from.

[pmichelli]

Hi David

SR 07336798

There is no 1GB anywhere. Both Data Domain are 10GB, connected to the same switch in the same datacenter. VMware and ESXi are all 10GBe uplinks. The only 1GB is the iDRAC.
Both appliances can backup single VMs at up to 300MB/sec and multiples in the same job at up to 800MB/sec
Both appliances can restore VMs at up to 250MB/sec just fine

I was moving a job that was 4TB and it took almost 12 hours to complete. Some VMs were moving data at 2MB/sec

The VBR server is the gateway for both the data domains. The only thing I can say is one DD is in one vLAN and the new one is in a secure vLAN (same vLAN where Veeam VBR lives)
The data we are trying to move is crossing vLANs. DD-OLD is in a vLAN outside of the VBR (legacy design). The new design everything lives in a hardened vLAN with only necessary ports to talk to what it needs to.

[david.domask]

Thank you for the case number and for the details and for checking. I think at this point let's allow the Support Engineers some time to review the logs and see what they spot.

[Spex]

Why not replicate the mtrees and scan/reimport/attach backups? Should be much faster and does not stress veeam servers. Of course it does not satisfy regarding your question...
Perhaps you can choose different gateways for the datadomains. Maybe this helps a bit.

[Robobeat]

I would highly recommend you to use the replication functionality built into the Data Domain.
We have several Data Domains and we always use the built-in replication since it is much faster and a lot more efficient.
This way (if handled correctly) the data will be replicated "as-is", keeping its ddboost capabilities like its deduplication and compression factor.
Using Veeam in a File copy or Back-up Copy job means Veeam will extract, uncompressing the data to its full size to then send it over and let the new Data Domain use deduplication and compression again.
This is most likely also the reason you get only speeds like 100MB/s.

I would advise to check the Data Domain Administration guide of any information on how to set it up.
Be aware to apply the DDBoost account to the newly created Mtree (after setting up the replication) to change it into a DDBoost Mtree.
Otherwise the DDBoost functionality will not be applied.
Let me know if you need any help on that.

Greets,
B. ten Velden

[david.domask]

Appreciate the advices Spex and Robobeat.

>Using Veeam in a File copy or Back-up Copy job means Veeam will extract, uncompressing the data to its full size to then send it over and let the new Data Domain use deduplication and compression again.
This is most likely also the reason you get only speeds like 100MB/s.

I don't disagree that mtree replication is an option, however, I would disagree with the conclusion on this statement -- many clients are using Veeam and DataDomain for the same operations without such a speed limit, so, barring a problem on the DataDomain side with dehydration or the DDBoost library, I suspect the issue resides elsewhere. The investigation is still on-going and we will see what the results are.

[michele.berardo]

Hi pmichelli,
I've done your same migration a couple months ago, so I perfectly know your situation and the limits you are encountering. Our migration takes more than one month instead of usual 1 week more-less because of these DD limitations. I've done a lot of investigation and raising a case with Dell EMC and the result is this is a read limitation of DD. So even if your backup job runs at 300MB/s (like our environment) and you have all front end interfaces at 10Gbps, read operation is very slow because of disks. Write operations use cache tier so the performance are totally different.
However replicating MTree remains the best option, because in the meantime you can run your daily jobs, but there is another limitation, if you increment your data day-by-day, your MTree replication will take more time to complete. Because of that, we have decided to create a temporary MTree, not replicated, on which run all the jobs during the migration.
During our migration, we have moved some Backup Repository content between different MTree using Veeam copy/move feature. Honestly I think Veeam rehydrates all data it reads because in this case reading operation was very very slow but it is in some way expected.

[aneo_cbu]

DD ist optimized on doing a lot of read-ahead operations and the usage of multiple streams for a single job. Unfortunately this is not very comaptible with the Filelayout Veeam is using for their backupdata.

So use mTree replication for the duplication of the data, will be a lot faster.

[fb77723]

Bigger picture, in a disaster recovery scenario where it's necessary to restore most of or an entire environment, the data domain read performance limitations would be a significant problem. Imagine a scenario where it takes a month or more to restore your environment. For day to day backups and small operational restores, DD is probably fine. For DR, it's not. It's difficult to beat the DD data reduction ratio but at what cost.

[gnalley]

One could also call this thread "'Insert backup/restore product name here' restore of backups really slow"... I challenge everyone on this thread who are using data deduplication / compression appliances to determine what their actual restore rate is and extrapolate that to the restoring their entire environment (if that is how you plan to recover from a complete disaster/ransomware event). Also, don't assume that everything will restore correctly the first time...assume a 10-25% failure rate. Does that model meet your organization's expectation(s) for recovery? I also recommend the same exercise if you are using cloud storage...imagine your cloud provider is using data deduplication/compression appliances that are shared with dozens of customers all processing jobs, and you have the latency of coming across the internet...The exercise could be eye opening.

[gnalley]

As you can tell from my earlier post, I also would disagree with the conclusion on this statement -- "many clients are using Veeam and DataDomain for the same operations without such a speed limit, so, barring a problem on the DataDomain side with dehydration or the DDBoost library, I suspect the issue resides elsewhere. The investigation is still on-going and we will see what the results are."

I would state that many customers are using Veeam and DataDomain for backup operations without such a speed limit, but the number that are using (insert backup software name here) and DataDomain for restores without such speed limits are *significantly* less.

[pber]

To add to what fb77723 commented on restore times. We had Data Domains for our Veeam backups and we had a major outage and had to restore many machines from Data Domain, the restore times were an eye opener that ultimately made us re-design our setup. Firstly, to restore directly from Data Domain will be too slow. We found it was much, much faster to copy the files off the data domain and to a Windows file server then do the restore from the Windows file server. The big thing is to get the files you are restoring off of the dedupe appliance first before restoring. It will be orders of magnitude faster. That alone probably saved us weeks. We've since moved to different vendor's dedupe appliances, although faster. Similar issue.

Regarding my re-design: We now keep a week worth of backups on fast SAN disk, then we do backup copy jobs to the slower dedupe/immutable appliances. I always have fast restores unless I have to go to a backup older than a week, but I work in a "real-time" environment that suits this design and I rarely ever have to go to the backups on the dedupe appliances.

[Matts N]

A quick question: you mention this goes through a router/firewall. Is that component capable of transmitting data at 10Gbps? Could be what is limiting traffic.

[igoradsilva]

Hi! Perhaps a little late, but DD has an option to tune the trade-off between sequential and random I/O operations. I think they created this option to adjust the system for read/restore operations, especially during disaster recoveries, where there's a huge amount of data that you need to read randomly. You can find this option on "Data Mangement"/"File System", then click on "Settings" and then on the "Workload Balance" tab.

And I agree that the best option to migrate from one DD to another is to use the appliance's replication functionality.

[nvdwansem]

Going to follow this one wondering what the outcome will be.

[mfbu]

[not using Data Domain myself]

This may not be specific to Data Domain.

We have a support case open (07568950) where moves between XFS based repositories take several days to complete, with hours of zero I/O while veeamagent process keeps burning 100% CPU (single-core).

I can only guess this to be because Veeam tries to fully re-calculate dedup (XFS fast cloning in our case) on the fly. For large VMs / lots of restore points, this will take quite some time.