Host-based backup of VMware vSphere VMs.
Post Reply
HendersonD
Expert
Posts: 158
Liked: 8 times
Joined: Jul 23, 2011 12:35 am

Slow restore speeds, why?

Post by HendersonD »

We backup all VMs from our production data center to our disaster recover site. We use a physical proxy server for these backups and leverage backup from storage snapshots using integration with our Nimble arrays. The two Nimble arrays, the physical proxy, and switches all have 10 gig connections. The backup jobs typically have a processing rate between 600 and 1,000 MB/s

Yesterday I did a restore of an entire Linux VM called Barracuda. The restore was successful but I noticed this
11/3/2020 9:16:21 PM Restoring Hard disk 1 (50 GB) : 45.7 GB restored at 69 MB/s [nbd]

Why are restores so slow? Any way to speed this up?
Gostev
Chief Product Officer
Posts: 31459
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Slow restore speeds, why?

Post by Gostev »

Please include the support case ID for this issue, as requested when you click New Topic. Thanks!
HendersonD
Expert
Posts: 158
Liked: 8 times
Joined: Jul 23, 2011 12:35 am

Re: Slow restore speeds, why?

Post by HendersonD »

Sorry about that
Case # 04477104
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Slow restore speeds, why?

Post by foggy »

What is the backup target in this setup? Did you perform the remote restore back to the production site?
HendersonD
Expert
Posts: 158
Liked: 8 times
Joined: Jul 23, 2011 12:35 am

Re: Slow restore speeds, why?

Post by HendersonD »

In production we have an all flash Nimble array. In our DR site we have a brand new Nimble HF40 hybrid array. There is a 10 gig connection between these two sites and the Nimble arrays, proxy server, and switches all have 10 gig connections. To answer your question the backup target is the hybrid array.

The VM we restored was sitting on a Veeam volume called VeeamBackup which comes from the hybrid array. This is mounted via iSCSI to the physical Veeam proxy server. We restored it back to our production data center with the data landing back on the all flash Nimble array
HendersonD
Expert
Posts: 158
Liked: 8 times
Joined: Jul 23, 2011 12:35 am

Re: Slow restore speeds, why?

Post by HendersonD »

With this being shown on the restore job, the restore happens via network mode
11/3/2020 9:16:21 PM Restoring Hard disk 1 (50 GB) : 45.7 GB restored at 69 MB/s [nbd]

What exactly is the path for the data when it is restored via network mode? Just trying to figure out if we have a 1 gig connection anywhere in this path
nitramd
Veteran
Posts: 297
Liked: 85 times
Joined: Feb 16, 2017 8:05 pm
Contact:

Re: Slow restore speeds, why?

Post by nitramd »

Have you considered using iPerf3 to help determine of you have a 1Gb link in the path?
HendersonD
Expert
Posts: 158
Liked: 8 times
Joined: Jul 23, 2011 12:35 am

Re: Slow restore speeds, why?

Post by HendersonD »

I could do that I just need to know when a restore job is running what is the source and target. In other words I have the raw data sitting on the Nimble array in my DR site and in the end it is being restored to the Nimble array in the production site. There are several different paths between these the start and the end, just need to know more about what path the data actually takes during a restore
backupquestions
Expert
Posts: 186
Liked: 21 times
Joined: Mar 13, 2019 2:30 pm
Full Name: Alabaster McJenkins
Contact:

Re: Slow restore speeds, why?

Post by backupquestions »

I believe the management nic interface of the esxi host will be used and so make sure that itself is 10gb.
HendersonD
Expert
Posts: 158
Liked: 8 times
Joined: Jul 23, 2011 12:35 am

Re: Slow restore speeds, why?

Post by HendersonD »

So when I restore from my Nimble array in my DR site with the data landing on the Nimble array in my production data site, the data passes through the management interface of the ESXi hosts in my production data center? Why would that be the case? My proxy server has
Direct access to the source data for a restore since it has a mounted drive from the Nimble array in production
Direct access to the Nimble array in production as well
So you are thinking the proxy server reads the source data at 10 gig then sends the data out the management interface of the proxy server to the management interface of an ESXi host in production?
PetrM
Veeam Software
Posts: 3229
Liked: 520 times
Joined: Aug 28, 2013 8:23 am
Full Name: Petr Makarov
Location: Prague, Czech Republic
Contact:

Re: Slow restore speeds, why?

Post by PetrM »

Hello,

I would say that this is definitely true based on this line:

Code: Select all

"1/3/2020 9:16:21 PM Restoring Hard disk 1 (50 GB) : 45.7 GB restored at 69 MB/s [nbd]"
I would recommend to clarify with our support team why NBD mode was selected instead of SAN as long as all requirements for SAN mode are met.
One more idea is to run restore in HotAdd mode, you will need to have virtual proxy and select the proxy at this step of the restore wizard (pick proxy to use).

Thanks!
HendersonD
Expert
Posts: 158
Liked: 8 times
Joined: Jul 23, 2011 12:35 am

Re: Slow restore speeds, why?

Post by HendersonD »

I did run a restore with hotadd and got 194 MB/s so that is triple the speed of nbd mode. Still not sure why SAN mode is not chosen for the restor
ChuckS42
Expert
Posts: 189
Liked: 27 times
Joined: Apr 24, 2013 8:53 pm
Full Name: Chuck Stevens
Location: Seattle, WA
Contact:

Re: Slow restore speeds, why?

Post by ChuckS42 »

One thing to consider is that when you restore from that hybrid Nimble, you're likely doing "cold reads" directly from the SATA disks; that'll slow things down a bit. Do you get Load results from a restore job? (e.g., Load: Source 63% > Proxy 73% > Network 54% > Target 0%)

Also - how loaded is the 10Gb link between sites? Do other systems use it?
Veeaming since 2013
HendersonD
Expert
Posts: 158
Liked: 8 times
Joined: Jul 23, 2011 12:35 am

Re: Slow restore speeds, why?

Post by HendersonD »

I learned from support that during a restored the ESXi management interface is used on the host where the VM is restored to. Currently our production ESXi hosts have 10 gig interfaces for VM, iSCSI, and vMotion traffic but only 1 gig interfaces for management. For ESXi a 1 gig interface for management is fine since the only traffic is the web interface. During Veeam restores though, this 1 gig interface is the pinch point. I did do a restore with hot add mode and got 195 MB/s which is triple the nbd mode. We can either use hot add mode for restores or look into getting our ESXi management interfaces up to 10 gig.

I was hoping restores used SAN mode which is used for backups
PetrM
Veeam Software
Posts: 3229
Liked: 520 times
Joined: Aug 28, 2013 8:23 am
Full Name: Petr Makarov
Location: Prague, Czech Republic
Contact:

Re: Slow restore speeds, why?

Post by PetrM »

I'd suggest to continue working with our support team to understand why Direct SAN mode is not used during restore.

Thanks!
HendersonD
Expert
Posts: 158
Liked: 8 times
Joined: Jul 23, 2011 12:35 am

Re: Slow restore speeds, why?

Post by HendersonD »

I was told by someone from the support team that nbd or hot add is used during restores and san mode is never used for restores. Is that not correct
PetrM
Veeam Software
Posts: 3229
Liked: 520 times
Joined: Aug 28, 2013 8:23 am
Full Name: Petr Makarov
Location: Prague, Czech Republic
Contact:

Re: Slow restore speeds, why?

Post by PetrM »

Thick-provisioned disks can be restored in Direct SAN mode, you may refer to this page on our help center. Also, please double check all requirements for SAN mode here.

Furthermore, keep in mind that you always have a possibility to escalate the support case.

Thanks!
HendersonD
Expert
Posts: 158
Liked: 8 times
Joined: Jul 23, 2011 12:35 am

Re: Slow restore speeds, why?

Post by HendersonD »

All of my VMs are thin provisioned so it looks like nbd and hot-add mode are the only methods available, correct?
Why isn't SAN mode available for thin provisioned disks?
Gostev
Chief Product Officer
Posts: 31459
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Slow restore speeds, why?

Post by Gostev »

Correct.

This is because Direct SAN transport mode does at best a few MB/s while restoring thin disks due to an architectural limitation in vSphere. There are existing discussions about this here from many years ago, in case you are interested in more details.
HendersonD
Expert
Posts: 158
Liked: 8 times
Joined: Jul 23, 2011 12:35 am

Re: Slow restore speeds, why?

Post by HendersonD »

Thanks for the response everyone. We know now that when doing a restore we will choose hot-add mode
PetrM
Veeam Software
Posts: 3229
Liked: 520 times
Joined: Aug 28, 2013 8:23 am
Full Name: Petr Makarov
Location: Prague, Czech Republic
Contact:

Re: Slow restore speeds, why?

Post by PetrM »

Hello,

By the way, there is an option to select "Restore to a new location, or with different settings" restore mode and convert disks to the thick format in this step of the restore wizard.
I'd suggest to select "Thick (eager zeroed)" type, it might be useful if you would like to compare SAN and HotAdd mode processing rates.

However, creating eager zeroed disks takes longer because zeroes are written to the entire disk therefore the duration of restore process can be anyway increased.

Thanks!
Post Reply

Who is online

Users browsing this forum: DataDefender and 82 guests