Slow restore speeds, why?

[HendersonD]

We backup all VMs from our production data center to our disaster recover site. We use a physical proxy server for these backups and leverage backup from storage snapshots using integration with our Nimble arrays. The two Nimble arrays, the physical proxy, and switches all have 10 gig connections. The backup jobs typically have a processing rate between 600 and 1,000 MB/s

Yesterday I did a restore of an entire Linux VM called Barracuda. The restore was successful but I noticed this
11/3/2020 9:16:21 PM Restoring Hard disk 1 (50 GB) : 45.7 GB restored at 69 MB/s [nbd]

Why are restores so slow? Any way to speed this up?

[Gostev]

Please include the support case ID for this issue, as requested when you click New Topic. Thanks!

[HendersonD]

Sorry about that
Case # 04477104

[foggy]

What is the backup target in this setup? Did you perform the remote restore back to the production site?

[HendersonD]

In production we have an all flash Nimble array. In our DR site we have a brand new Nimble HF40 hybrid array. There is a 10 gig connection between these two sites and the Nimble arrays, proxy server, and switches all have 10 gig connections. To answer your question the backup target is the hybrid array.

The VM we restored was sitting on a Veeam volume called VeeamBackup which comes from the hybrid array. This is mounted via iSCSI to the physical Veeam proxy server. We restored it back to our production data center with the data landing back on the all flash Nimble array

[HendersonD]

With this being shown on the restore job, the restore happens via network mode
11/3/2020 9:16:21 PM Restoring Hard disk 1 (50 GB) : 45.7 GB restored at 69 MB/s [nbd]

What exactly is the path for the data when it is restored via network mode? Just trying to figure out if we have a 1 gig connection anywhere in this path

[nitramd]

Have you considered using iPerf3 to help determine of you have a 1Gb link in the path?

[HendersonD]

I could do that I just need to know when a restore job is running what is the source and target. In other words I have the raw data sitting on the Nimble array in my DR site and in the end it is being restored to the Nimble array in the production site. There are several different paths between these the start and the end, just need to know more about what path the data actually takes during a restore

[backupquestions]

I believe the management nic interface of the esxi host will be used and so make sure that itself is 10gb.

[HendersonD]

So when I restore from my Nimble array in my DR site with the data landing on the Nimble array in my production data site, the data passes through the management interface of the ESXi hosts in my production data center? Why would that be the case? My proxy server has
Direct access to the source data for a restore since it has a mounted drive from the Nimble array in production
Direct access to the Nimble array in production as well
So you are thinking the proxy server reads the source data at 10 gig then sends the data out the management interface of the proxy server to the management interface of an ESXi host in production?

[PetrM]

Hello,

I would say that this is definitely true based on this line:

Code: Select all

"1/3/2020 9:16:21 PM Restoring Hard disk 1 (50 GB) : 45.7 GB restored at 69 MB/s [nbd]"

I would recommend to clarify with our support team why NBD mode was selected instead of SAN as long as all requirements for SAN mode are met.
One more idea is to run restore in HotAdd mode, you will need to have virtual proxy and select the proxy at this step of the restore wizard (pick proxy to use).

Thanks!

[HendersonD]

I did run a restore with hotadd and got 194 MB/s so that is triple the speed of nbd mode. Still not sure why SAN mode is not chosen for the restor

[ChuckS42]

One thing to consider is that when you restore from that hybrid Nimble, you're likely doing "cold reads" directly from the SATA disks; that'll slow things down a bit. Do you get Load results from a restore job? (e.g., Load: Source 63% > Proxy 73% > Network 54% > Target 0%)

Also - how loaded is the 10Gb link between sites? Do other systems use it?

[HendersonD]

I learned from support that during a restored the ESXi management interface is used on the host where the VM is restored to. Currently our production ESXi hosts have 10 gig interfaces for VM, iSCSI, and vMotion traffic but only 1 gig interfaces for management. For ESXi a 1 gig interface for management is fine since the only traffic is the web interface. During Veeam restores though, this 1 gig interface is the pinch point. I did do a restore with hot add mode and got 195 MB/s which is triple the nbd mode. We can either use hot add mode for restores or look into getting our ESXi management interfaces up to 10 gig.

I was hoping restores used SAN mode which is used for backups

[PetrM]

I'd suggest to continue working with our support team to understand why Direct SAN mode is not used during restore.

Thanks!

[HendersonD]

I was told by someone from the support team that nbd or hot add is used during restores and san mode is never used for restores. Is that not correct

[PetrM]

Thick-provisioned disks can be restored in Direct SAN mode, you may refer to this page on our help center. Also, please double check all requirements for SAN mode here.

Furthermore, keep in mind that you always have a possibility to escalate the support case.

Thanks!

[HendersonD]

All of my VMs are thin provisioned so it looks like nbd and hot-add mode are the only methods available, correct?
Why isn't SAN mode available for thin provisioned disks?

[Gostev]

Correct.

This is because Direct SAN transport mode does at best a few MB/s while restoring thin disks due to an architectural limitation in vSphere. There are existing discussions about this here from many years ago, in case you are interested in more details.

[HendersonD]

Thanks for the response everyone. We know now that when doing a restore we will choose hot-add mode

[PetrM]

Hello,

By the way, there is an option to select "Restore to a new location, or with different settings" restore mode and convert disks to the thick format in this step of the restore wizard.
I'd suggest to select "Thick (eager zeroed)" type, it might be useful if you would like to compare SAN and HotAdd mode processing rates.

However, creating eager zeroed disks takes longer because zeroes are written to the entire disk therefore the duration of restore process can be anyway increased.

Thanks!