Very slow restore performance from DataDomain

[lenart.plausteiner]

Hello folks. Help would be more then appreciated as we're very fustrated.

We're strugling with Veeam support team for more then 2 month now with no progress on our issue. And we're aware that restore speeds from PBBA device are slow (rehidration, decompress, block alligning, etc...), but not this slow.

We have environment with 15 x ESXi hosts and 250VM's which are backuped to the primary SAN storage and then transfered with copy job to the DataDomain. Backups and Backup Copy Jobs are working super fine with speeds up to 4Gbps. When we proceed with restoring from DataDomain we get speeds around 60-80MB/sec in a single restore job.

We've done some additional testing with DDUtil tool, as suggested from Veeam, we reach speed 150MB/sec or more. We can start multiple reads streams with DDUtil and we get combined speed as high as 300MB/sec. On the other side, we have also tried with multiple restore sessions within Veeam B&R where we also get higher combined speeds. My thought is that we're not reaching the botlneck of the DD device and somethin within Veeam components is preventig us for getting perfomance as expected.

We use for source Dell EMC DataDomain DD6300 with two expansion units. Device is connected to network with 10Gb interface with no bonding (single interface). DD is used only for Veeam (not shared) and only one MTree is configured. Our interface and the whole transport(End to End) path is configured with MTU 9000 (jumbo frames).

When we do a read test with DDUtils from server existing Veeam proxy server we get read speed 152MB/sec. This server is domain joined and connected to network with 10Gb/sec. Speed was: 152MB/sec, total read 139407 Mb total time elapsed 00:15:17

When we proceed with DDUtils on physical server(freshly installed) and connected with 1Gbps adapter we get this: 95MB/sec, total read 139407 Mb total time elapsed 00:24:25

When we proceed with DDUtils on physical server(freshly installed) and connected with 10Gbps adapter we get this: 142MB/sec, total read 139407 Mb total time elapsed 00:16:17

When we try with restore job within Veeam B&R we get this results 62MB/sec, total read 139407 Mb total time elapsed 00:48:13 (results are always the same, regardless which proxy is used).

Additional information regarding environment:
1 x Veeam B&R Server (10CPU's, 48GB RAM)
4 x Veeam proxy servers (each 16Gb RAM and 8 CPU's)
2 x Primary backup repository (SAN Attached block storage)
1 x DellEMC Datadomain DD6300

We've opened support ticket within Veeam. Our ticket ID is:05994180.

Any directions are appreciated.

[PetrM]

Hello,

First of all, I'd suggest to ask your engineer about the "bottleneck" shown in the debug logs. I'm pretty sure that read from the repository is the slowest operation in a data processing conveyor but it's always better to make sure. If this assumption was correct, I'd try to find out the difference between read mechanisms in DDutils and in Veeam Data Mover. I guess DDutils is the best tool for DataDomain performance testing but there are no tools that repeat Data Mover's logic with 100 % accuracy. I suppose DDutils just reads data blocks sequentially while read from .vbk performed by Data Mover can have some random read (the blocks of the same disk can reside in different sectors) or decompression, etc. I would say that it's normal that DDutils is a bit faster than our restore.

In this KB you will find the instructions to escalate a support request if you are not satisfied with how your case is being handled.

And the main question is: what is the ultimate goal of this research? What speed do you expect? Also, you may consider a dedupe appliance as a secondary target for backup copy and use another faster storage for the primary repository.

Thanks!

[Gostev]

Last month I celebrated 15 years of answering "Very slow restore performance from [insert a deduplication appliance name]" type of threads on this forum

You asked for a direction so here's my suggestion on what I personally would do.

Veeam will read backups as fast as your backup storage can serve the requested blocks, simple as that. So if you want fast restores, get a storage that can serve those blocks fast, without a ton of data processing in the way. You can easily get 1GB/s+ restores, which is 20 times faster than what you get now, by using a regular server with a bunch of disks as your primary repository. So much cheaper than your DataDomain - and if you format it with ReFS or XFS, you will also get some very decent (10-15x) data reduction ratio too.

While your DataDomain will make an awesome secondary repository for backup copies, with a longer-term retention (if this is something you require).

[Trelor]

lenart.plausteiner,

We actually used to use 2x DD6300s in our environment and after realizing that the restore speeds on the DD was a feature of the appliance we migrated to a FlashBlade. The data domain is great to write data to for compliance reasons and hope that you will never need that data again. The restores are painfully slow and most of the time we decided the data we needed wasn't worth the trouble.

[rennerstefan]

Additional Informations on the best practices with Deduplication Appliances can be found here: https://www.veeam.com/kb1745