Discussions specific to the VMware vSphere hypervisor
Post Reply
Redline
Novice
Posts: 5
Liked: never
Joined: Nov 12, 2015 5:13 am

SQL restore speed

Post by Redline » Mar 14, 2016 10:46 am

I have a problem with restore speed on my v9 installation. If I try to restore using restore wizard -> Application items -> SQL server and restore to some SQL server I get about 85 Megabit/sec speed over my 1Gb/s link. But if I choose Extort database and put it on a shared folder of that same SQL server I get 900 Megabit/sec. Simple copy using windows explorer from Veeam server to SQL also shows good speeds around 900 Megabits/s.

One thing I noticed is in SQL item-level restore scenario it mounts backup to the target server c:\VeeamFLR folder and then copies it to the destination folder on the same server (and getting bad network speed). As I understand at this point no SQL activity is involved, just basic file copy? So what could be the problem?

Image

foggy
Veeam Software
Posts: 18158
Liked: 1542 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: SQL restore speed

Post by foggy » Mar 14, 2016 11:44 am

Could you please describe your setup in a bit more detail - where Veeam B&R is itself installed, what kind of repository is involved? Also, what restore options exactly are used in both cases? Thanks.

Redline
Novice
Posts: 5
Liked: never
Joined: Nov 12, 2015 5:13 am

Re: SQL restore speed

Post by Redline » Mar 14, 2016 1:03 pm

My test lab has vcenter server and a couple of HP DL360 G7 and Gen8 hosts. Veeam B&R is installed in Win 2012 R2 VM on one of the hosts, SQL is Win 2008 R2 VM on another host. All hosts are physically in the same location and connected via Cisco 3750 switch.

As for backup repository - I have HP P2000 (FC) with 12 2TB NL-SAS drives in Raid-6, it has a LUN presented to all hosts and formatted as VMFS datastore.

For restore in first case I open Restore -> Application Items -> Microsoft SQL server -> choose a full backup and get to the SQL Explorer. There I choose one of the DB's, Database restore -> Restore to another server -> restore to the selected image-level backup point -> enter SQL server domain name and necessary login/pass -> choose folder on destination server. In this case I see network speed jumping all over the place 70-80-100 Mb/s but no more than 150 anyway.

Second time I open Restore -> Application Items -> Microsoft SQL server -> choose a full backup and get to the SQL Explorer. There I choose one of the DB's, Export -> Export to another folder -> restore to the selected image-level backup point -> choose the same shared folder as in the first case on destination server. Here I see network speed steady at 900-910 Mb/s

foggy
Veeam Software
Posts: 18158
Liked: 1542 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: SQL restore speed

Post by foggy » Mar 15, 2016 12:18 pm

In the first case, the backup is indeed mounted to remote server, while in the second it is mounted locally and file is copied over the network (similar to your "Simple copy using windows explorer" test).

What Veeam B&R version do you use?

Redline
Novice
Posts: 5
Liked: never
Joined: Nov 12, 2015 5:13 am

Re: SQL restore speed

Post by Redline » Mar 15, 2016 12:40 pm

Yes, I understand this, but what can be the reason for such a difference in the speed? I use v9.0.0.902 for this test.

On a side note, I've noticed in the similar test on v8 it is mounting backup to the local server no matter if I choose SQL item-level recovery or Export. This is a screenshot of my recovery from v8 (Restore -> Application Items -> Microsoft SQL server -> choose a full backup and get to the SQL Explorer. Choose DB, Database restore -> Restore to another server -> restore to the selected image-level backup point -> enter SQL server domain name and necessary login/pass -> choose folder on destination server.) And the speed is high in this case too
Image

Maybe as a workaround there is a way to force it to mount the backup locally in v9?

Gostev
SVP, Product Management
Posts: 24656
Liked: 3471 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: SQL restore speed

Post by Gostev » Mar 16, 2016 2:24 pm

I've talked to the devs and they are quite puzzled, and would like to investigate this further, possibly via webex. If you are available, please open a support case and let me know the case ID. Thanks!

Redline
Novice
Posts: 5
Liked: never
Joined: Nov 12, 2015 5:13 am

Re: SQL restore speed

Post by Redline » Mar 18, 2016 5:42 am

Hi, I've opened the case - #01734574

Redline
Novice
Posts: 5
Liked: never
Joined: Nov 12, 2015 5:13 am

Re: SQL restore speed

Post by Redline » Mar 29, 2016 8:08 am

I got the answer from support - as I understand the low speed in SQL restore operations is due to the fact that v9 is mounting backup to the target server to perform necessary operations (as opposed to v8 which mounts to the veeam server itself). And this new way of work was not yet properly optimized, so for now if you want to get full speed in SQL restore you should either use the Export option or start the restore from the Veeam Remote Console installed on the SQL server

eshteinman
Novice
Posts: 4
Liked: 2 times
Joined: Dec 09, 2014 10:54 am
Full Name: Evgeny Shteynman
Contact:

Re: SQL restore speed

Post by eshteinman » Jul 26, 2016 9:58 am

Are there any changes in this regard??
after the update 1 situation has not changed (((

foggy
Veeam Software
Posts: 18158
Liked: 1542 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: SQL restore speed

Post by foggy » Jul 26, 2016 10:58 am

No changes, as far as I know. I recommend you contacting support to confirm the issue and also for tracking purposes, since the most often reported issues get fixed first.

eshteinman
Novice
Posts: 4
Liked: 2 times
Joined: Dec 09, 2014 10:54 am
Full Name: Evgeny Shteynman
Contact:

Re: SQL restore speed

Post by eshteinman » Aug 08, 2016 8:58 am

after apply Update 2 no changes, restore speed stay very slow((( I don't understand why so important bug not fix

yunales
Influencer
Posts: 15
Liked: 4 times
Joined: Aug 20, 2014 7:34 pm
Full Name: P K
Contact:

Re: SQL restore speed

Post by yunales » Aug 08, 2016 11:23 am

Hello guys,

this thread got me curious so I tested the same with an environment of one of our customers.
And I can reproduce exactly the same issues with restore speed as the thread starter!

While using the " SQL restore to another servers" yields around 100-140 MBIT/s which is like 12-17 MB/s , the "export to another folder" get around 700-800 MBIT which is acceptable.

So please forward this issue to R&D this is a major falw. What good is a ultra fast backup of a 2 TB Database, when the restore takes 2 days to complete??

regards

foggy
Veeam Software
Posts: 18158
Liked: 1542 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: SQL restore speed

Post by foggy » Aug 08, 2016 1:33 pm

Have you tried to install Veeam B&R remote console on the target SQL Server itself, to allow local mount?

Vitaliy S.
Product Manager
Posts: 22874
Liked: 1540 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: SQL restore speed

Post by Vitaliy S. » Aug 08, 2016 4:45 pm

@Evgeny, @yunales, please open support cases for these issues, so that we could forward this issue to our dev team.

yunales
Influencer
Posts: 15
Liked: 4 times
Joined: Aug 20, 2014 7:34 pm
Full Name: P K
Contact:

Re: SQL restore speed

Post by yunales » Aug 09, 2016 7:58 am 2 people like this post

Hey Vitaly,

case is filed @01875015

regards

eshteinman
Novice
Posts: 4
Liked: 2 times
Joined: Dec 09, 2014 10:54 am
Full Name: Evgeny Shteynman
Contact:

Re: SQL restore speed

Post by eshteinman » Aug 10, 2016 8:23 am

Please write in this topic answer from technical support

JoshuaPostSAMC
Expert
Posts: 124
Liked: 17 times
Joined: Jul 30, 2015 7:32 pm
Contact:

Re: SQL restore speed

Post by JoshuaPostSAMC » Aug 15, 2016 3:18 pm

We just ran into this same thing. Restored a 100 GB database on a VM in our UCS cluster with 10 GB networking and it took almost 3 hours to complete. I'm collecting details of how the restore was ran, but I expect it was from a remote console on the admin's workstation and trying to restore directly into the target SQL server. It really needs to be addressed so the restores can happen as fast as possible.

yunales
Influencer
Posts: 15
Liked: 4 times
Joined: Aug 20, 2014 7:34 pm
Full Name: P K
Contact:

Re: SQL restore speed

Post by yunales » Aug 18, 2016 11:55 am

just a quick update. As requested by Vitaly I opened up a case with reference to this thread. But the case was not forwared to the Dev team. Instead I need to do some testing and upload the log bundle.

here's the answer from Roman Rakhimov:
I'm sorry to bother you.

My escalation engineer required more data to continue this case handling.


Can you please clarify, is restore with 'export' done locally or to the same target server?

Please provide us with following information:

1. Infrastructure description:
- Veeam B&R version and build number
- Backup job name used in SQL DB restore session
- Server name where console was executed for restore session
- SQL VM name
- Backup repository name where backup file is located (type of connection: NFS, iSCSI, CIFS, DAS)

2. Collect restore sessions logs:
* One bundle for 'restore to original' SQL server
* Second log bundle for 'export to another folder'
------------------------------------------------------------

I dont understand why I must test and log everything again since it's cleary a bug! But so be it....

Gostev
SVP, Product Management
Posts: 24656
Liked: 3471 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: SQL restore speed

Post by Gostev » Aug 18, 2016 9:58 pm

9.5 will have SQL Server database restore performance improved in a few times.

arosas
Enthusiast
Posts: 34
Liked: 6 times
Joined: Jun 09, 2015 9:33 pm
Full Name: Tony Rosas
Contact:

Re: SQL restore speed

Post by arosas » Apr 12, 2017 2:44 pm

9.5 still has the same speed issue. Has this been addressed or is installing the console on the target SQL servers the only solution?

foggy
Veeam Software
Posts: 18158
Liked: 1542 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: SQL restore speed

Post by foggy » Apr 12, 2017 2:50 pm

Hi Tony, yes database restore performance was improved in v9.5 up to 3 times. If you're still seeing performance issues, I suggest contacting support for a closer look at your setup (reasons could be environmental).

arosas
Enthusiast
Posts: 34
Liked: 6 times
Joined: Jun 09, 2015 9:33 pm
Full Name: Tony Rosas
Contact:

Re: SQL restore speed

Post by arosas » Apr 24, 2017 4:59 pm 1 person likes this post

We have all SQL backups going to a certain repository. Support suggested changing the mount server setting for this repository to always be our SQL staging server instead of the Veeam server. When we did this the restore speed was tremendously better.

acoman
Lurker
Posts: 2
Liked: never
Joined: Sep 05, 2014 6:27 pm
Full Name: Adrian Coman
Contact:

Re: SQL restore speed

Post by acoman » Oct 17, 2017 1:41 pm

Hello,

I have the same problem with a SQL DB restore using the command shell of Veeam Explorer for Microsoft SQL Server, and thanks to veeam-backup-replication-f2/automate-re ... ml#p237865 who provided with a nice script that can be used either with Windows task scheduler, or just configured to run at the end of the backup job.
My problem is that to restore a 10GB DB from a Nimble Storage repository takes around 3 hours.....

Did someone find any solution to improve the restore speed?

I forgot to mention I use the lates version of Veeam B&R 9.5.0.1038, on which as per Gostev vmware-vsphere-f24/sql-restore-speed-t3 ... ml#p185375 the speed transfer has been improved.

Regards

foggy
Veeam Software
Posts: 18158
Liked: 1542 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: SQL restore speed

Post by foggy » Oct 18, 2017 11:39 am

Hi Adrian, I suggest you contacting support to verify whether your setup is optimal in terms of restore and identify the performance bottleneck. Btw, do you restore from a storage snapshot?

Post Reply

Who is online

Users browsing this forum: No registered users and 11 guests