Poor Performance Accelerated Restore with Dell Data Domain or Dedup Appliances in generall

[mtrimmel]

Hi Gostev,

I try to post this here in the hope u can pick this topic up and give an answer directly from your side.

We have multiple customers who use Veeam in combination with Dell Data Domain. Those customers have good restore performance (~ 400 - 500 MB/sec) when they restore an entire VM on Hyper-V or VMware because of the Veeam Feature Accelerated Restore. But a lot of enterprise customers are also using Agents like Windows, Linux, Oracle RMAN, SAP HANA as well DR from On-Premise to Azure. For all those agents the Accelerated Restore is not working and the performance drops by 10. So the performance drops from ~ 400 - 500 MB/sec to ~40-60 MB/sec. See here the documentation of the performance vaules https://drive.google.com/file/d/14TICPz ... sp=sharing

Why is this feature not implemented for all agents? I really strong believe in VEEAM, we also know the benefits of Dell Data Domain. I know the ultimate backup architecture and the 3-2-1 rule, but not all customers are willing to buy that big hardware footprint and look for true airgap with the Dell Cyber Recovery Architecture .

Other venders like i.e. Veritas NetBackup have implemented something similar to the accelerated restore feature, but they have this for years for all agents.

We tried to adress this with the local veeam team and veeam support, but with no results. Veeam Case #05725911, #06080375,#06143863

Is there any chance that Veeam will implement the accelerated restore feature for all agents including azure?

Would be very kind to get your opinion on this topics, regards Markus

[Andreas Neufert]

In general our restore works up to the speed of the storage for the specific load pattern that you send to the storage.

For VM based restore, we read sequentially (and write randomly to VMware) to accelerate with read ahead caching usage on the data domain side.

For Oracle RMAN and SAP HANA Plug-in backup/restore we sequentially read/write with the datadomain and therefore read ahead caching works correctly there.

For InGuest File and Application restore as well as for Instant VM Recovery the read pattern from the data domain is purely random IO and there the storage just do not shine as it was not built for random IO processing.

For Direct Restore to Azure, this is sequential read and so the read ahead caching is working as well. But usually the upload is limited on the Azure side (limitation outside of Veeam).

For any operation we have (even not visible in the UI) in the logs a bottleneck analyze and we check your support cases on where the bottleneck really is. Let me check with the team. Thanks

[Andreas Neufert]

In the meantime, do you run latest supported Veeam and DataDomain version. Over time we did a lot of enhancements on both sides.
https://www.veeam.com/kb4420
and DD OS 7.10

[mtrimmel]

Andreas,

thank you very much for your quick and detailed answer. U know more than support does...., but I am really happy about your explanation.

If I understand you correctly the sequential read is working for azure, right?

We had a long support ticket, which got closed with the statement not supported. Veeam Case #05725911

How can I explain this to support? In our environment it’s not working with azure.

Does your explanation mean as well that the Dell Data Domain MSR Restore feature is working for azure restores? https://www.dell.com/support/kbdoc/en-k ... -and-later

Are there any plans to improve "InGuest File and Application" restores?

regards
Markus

[Andreas Neufert]

Let me check a bit closer regarding the Azure restore scenario as I got some information related to this that it needs a specific configuration to work.

Dell gave us through the API the control if read ahead cache is used or not. Read ahead cache enablement makes only sense for sequential workstreams. For random work streams like File Level restore or any Guest File or Instant restore it causes only overhead and slows down the processing as the backend read way more data from the disks as needed (4-5x more IO on the disk backend just to throw it away as the read is random). So we can not accelerate for example file level recovery with this.

[mtrimmel]

Andreas,

currently we are at veeam v11, should we realy upgrade to 12? Will this make any differences from your opinion? DDOS is on 7.10.

regards
Markus

[Andreas Neufert]

It is really hard to say. v12 latest version in general has like 1-2 years of additional improvements in the product. Maybe do a test with a separate v12 latest version install and a separate storage on your DataDomain.

[mtrimmel]

Andreas,

we will just upgrade to v12 and give you feedback. Would be great if you could check again the azure stuff. This is very important for a customer of us.

Add)

In general our restore works up to the speed of the storage for the specific load pattern that you send to the storage.

For VM based restore, we read sequentially (and write randomly to VMware) to accelerate with read ahead caching usage on the data domain side.

For Oracle RMAN and SAP HANA Plug-in backup/restore we sequentially read/write with the datadomain and therefore read ahead caching works correctly there.

For InGuest File and Application restore as well as for Instant VM Recovery the read pattern from the data domain is purely random IO and there the storage just do not shine as it was not built for random IO processing.

For Direct Restore to Azure, this is sequential read and so the read ahead caching is working as well. But usually the upload is limited on the Azure side (limitation outside of Veeam).

For any operation we have (even not visible in the UI) in the logs a bottleneck analyze and we check your support cases on where the bottleneck really is. Let me check with the team. Thanks

[Andreas Neufert]

Other question. For the Azure upload. Did you tested as well another storage type?
Backup the VM again to a Windows Repository. Then start Direct Restore to Azure from there just to to be sure that we are searching at the right place for optimization.

[mtrimmel]

Hi,

we have the following concept. On-Premise Backup to Dell Data Domain using VBR on Hyper-V. Dell mtree Replication to Data Domain Virtual Edition on Azure (250 TiB uses Azure Blob Backend, 4 Systems). In a DR case we install a new VBR Server on Azure, add the repositories from the Data Domain Virtual Edition. Restore to Azure. Reason for this is the required network bandwith from On-Premise to Azure. Dell mTree replication is 4 times more efficent (less network traffic) than VBR directly to Azure Blob.

Basically we have the setup from figure 2 in this link. https://www.dell.com/support/kbdoc/en-u ... datadomain

We did some benchmarking with Data Domain virtual Edition having data on SSD on Azure, there we are able to restore with ~ 1000 MByte/sec with 10 Restores with VBR. But this cant be used for production, because its to expensive as well not supported by Dell.

With the regular setup (DDVE 250 TiB using azure Blob Backend) we see in the logs that Dell MSR is not used, as well the restore performance is very bad. 5-25 MB/s per Disk, support told us this is because accelerated restore is not supported on azure.

regards Markus

[Andreas Neufert]

OK thanks. Let me check about the MSR usage.

[Andreas Neufert]

Can you maybe do a small test.
Can you create a repository on a cloud VM and copy one of the backups there (maybe small VM).
Then do the Direct Restore to Azure from there to have a baseline comparison speed.

Are you sure that with the DDVE in Azure usage, that you do not pull down the data to a onpremises server before going into the cloud again? Mount server and cloud gateway server in the cloud configured? Console as well,...

[mtrimmel]

Hi,

the DDVE for sure uses the right gateway server. Mount Server should be relevant for boost restores, but thats set correctly as well. We did some tests from the DDVE to null, so we exposed the veeam boost repo on the data domain via NFS to a Linux Host on Azure and copied to file to null. There we get ~ 150 MB/sec per copy and a total of 600 MB/sec. Thats the baseline what we try to get with veeam, for us it seems that the accelerated restore is either not used or not working correctly in our environment.

regards
Markus

[Andreas Neufert]

Try to baseline the Direct Restore to Azure as well. Not that we search at the wrong place.

You can maybe try to use the exported backup files for this.

[mtrimmel]

Hi Andreas,

I hope you are doing well. Thank you for your recommendations, but I think its the wrong place to solve the issue. For us it would be important to clarify the product support question. Is it possible that those informations from you are documented somewhere official? Think this link should be updated with your infromations.

https://helpcenter.veeam.com/docs/backu ... ml?ver=120

Code: Select all

For VM based restore, we read sequentially (and write randomly to VMware) to accelerate with read ahead caching usage on the data domain side.

For Oracle RMAN and SAP HANA Plug-in backup/restore we sequentially read/write with the datadomain and therefore read ahead caching works correctly there.

For InGuest File and Application restore as well as for Instant VM Recovery the read pattern from the data domain is purely random IO and there the storage just do not shine as it was not built for random IO processing.

For Direct Restore to Azure, this is sequential read and so the read ahead caching is working as well. But usually the upload is limited on the Azure side (limitation outside of Veeam).

For any operation we have (even not visible in the UI) in the logs a bottleneck analyze and we check your support cases on where the bottleneck really is. Let me check with the team. Thanks

regards
Markus

[foggy]

Hi Markus, let me add a bit of context here. First, there are two separate features/optimizations:

- DDBoost read-ahead cache that Veeam B&R can enable/disable for certain workloads during restore depending on the feasibility (enable for sequential workloads, disable for random I/O like in IR/FLR).
- Accelerated Restore feature in Veeam B&R that is currently used for the full VM restores to VMware/Hyper-V only.

So Accelerated Restore is not currently enabled for restores to Azure nor it is used for agents and enterprise plug-ins. We see the demand and are discussing the ability to enable it for other restore workloads in the future. Thanks for the detailed feedback!

[mtrimmel]

Hi Foggy,

Sorry but I don't understand your posting, can please explain me?

What does this feature mean congrete?

DDBoost read-ahead cache that Veeam B&R can enable/disable for certain workloads during restore depending on the feasibility (enable for sequential workloads, disable for random I/O like in IR/FLR).

When is the restore than sequential from did perspective?

Will this mean that Dell MSR should work?

https://www.dell.com/support/kbdoc/en-i ... -and-later

For which agents will this work and what should it bring performance wise?

Regards Markus

[foggy]

Will this mean that Dell MSR should work?

Yes, Dell MSR works for sequential reads for large files (over 8GB), for full VM restores, for example.

For which agents will this work and what should it bring performance wise?

To my understanding, read-ahead cache on Data Domain should work similarly for all Veeam products but I need to check this internally.

[mtrimmel]

Thx foggy, sorry to ask again

Yes, Dell MSR works for sequential reads for large files (over 8GB), for full VM restores, for example

I understood from your previous posts, that only with the accelerated restore feature the read is done sequentially.

The Dell MSR feature only kicks in when the size is more than 8 GB and the read is sequential .

But isn't the veeam read during a restore always random from the data domain perspective, except accelerated restores?

Would be great if u could explain.

Thx Markus

[Andreas Neufert]

You mean because of the dedup engine and how it stores data on the storage.
Sequential in case of DD means that you read the data in the same way they are stored in a single file. That way the dedup engine knows exactly where the next data is read from and can for example use the read ahead cache to store already the next blocks in memory.

As explained above we store our data from multiple VM disks "randomly in (~100MB) chunks" in the backup file. So if you for example transfer a VMDK file into the cloud (converter) it is semi sequential. In case of VM restore we read the data truly sequential from DD but write the data randomly to many VM disks. This is as the primary storage can handle random writes faster than data domain can deliver data sequentially.

[Andreas Neufert]

Anyway coming back to my recommendation above. My point is the data domain read (just reading the file) is faster than the restore process. So my thinking is that there is another bottleneck somewhere in the Direct to cloud restore process. I think you are the customer that works with our field on it and I advised them on my suggestions above to verify the bottleneck. They can have a look at the logs the the more enhanced bottleneck analyze as well.

[mtrimmel]

Andreas, I am working on this. Update u asap

[JRRW]

Interesting thread. Clearly, DD has improved, as the last time I used that solution it was an absolute nightmare on performance for anything. And don't even try to run a health check on backups.

I'll admit that it's of some interest in regards to an alternative to tape for long term backup; we're also considering Isilon as it does dedupe.

Best of luck.

[mtrimmel]

Hi Andreas and Foggy,

it took me a while to do the testing. The restore via ddboost is around 3 times slower than via nfs with the same data from the same data domain.

Please have a look to the following document, for me it looks there is an issue in the ddboost restore implementation of VBR.

Here the documetation of the results and how to reproduce it. It can be reproduced with Hyper-V and VMware Backups.

https://drive.google.com/file/d/1nleOeZ ... sp=sharing

In addition the Excel with the parsed performance data.

https://docs.google.com/spreadsheets/d/ ... ue&sd=true


will also test azure blob native and give an update on this.

regards
Markus

[foggy]

Hi Markus, thanks for such a detailed report. I have one thought off the top of my head - in DDboost SDK 7.7 (included with v12), Dell modified authentication-related code to support FIPS in DDOS 7.2, which resulted in significant connection degradation. We've observed much slower connection establishment (~20x times slower compared to DDBoost SDK 7.0 included with v11a) which may result in an overall slower restore as well.

To negate the performance penalty, we've implemented certain optimizations around connection reuse in v12a (and will work on a global connection cache in v13). So to eliminate that factor, tests with either v12a or v11a are needed.

[aneo_cbu]

Hi Andreas and Foggy,

i've done the test with 11a and updated the doc and excel below. Performance is slightly better but still slower than doing the restore via NFS.

doc: https://quorumconsultinggmbh-my.sharepo ... Q?e=GnIhOO
excel: https://quorumconsultinggmbh-my.sharepo ... Q?e=gz2cje

regards

Christian

[heinrichloewen]

Hello,

my own tests also shows me that a copy to or from a Data Domain is much faster with NFS/CIFS than with Boost.

Copy/backup to a Data Domain with Boost takes the load to the cpus from the gateway and the task only use/take one cpu per stream. (Faster cpus -> faster transfer)
Copy/backup to a Data Domain with NFS/CIFS takes the load to the cpus from the Data Domain and use more than one cpu on the Data Domain.

If we backup/copy much data to the Data Domain over many gateways with Boost, it's better for throughput instead of using NFS/CIFS.

The only thing I didn't understand until now is, why is copy/restore from Data Domain so slow with Boost and fast with NFS/CIFS.

regards

Heinrich

[aneo_cbu]

hi,

the links to the v1.1 timed out, so here the same with google drive. Enjoy
PDF v1.1 - https://drive.google.com/file/d/15xoI_s ... sp=sharing
Excel v1.1 - https://docs.google.com/spreadsheets/d/ ... ue&sd=true