Extremely Long Restore times using VEEAM Explorer for MS SQL

[Brandon0830]

I am trying to see if someone can potentially shed some light onto some issues we’ve been having regarding SQL Restores using the VEEAM Explorer for MS SQL. VEEAM is restoring the databases successfully but they are taking absolutely forever. It’s been so bad that our DBA’s ended up having to ditch VEEAM in lieu of traditional SQL backups because they can restore from the same storage I have set up as my VEEAM repository 5X faster. It’s a shame because our DBA’s said it’s the nicest interface for restores he’s ever seen and was a major factor in the reason we bought VEEAM to begin with.

I have done tons of trial and error and the only thing that I can come up with is that there is something going on with the Staging Server or iSCSI connection during the restore.

I tested the “Export Databases” option with a 200GB database that didn’t have to do a point-in-time restore which just mounted the backup to C:\VeeamFLR on the VEEAM server and chose the option to export to a different server via UNC path \\XX\c$….and it was way faster than what I’ve been seeing (41 Minutes). Once I tested the “Export Databases” option with a point-in-time restore of the same exact database which forces it to use the Staging Server/iSCSI connection, it took forever exactly like we’ve been seeing (2 Hours and 41 Minutes). I’m assuming that this is the same thing that’s causing it to be so slow for us when we choose the native database restore option as well (e.g. 300GB Database has taken 5-6 hours to restore with VEEAM. Our DBA’s can restore with their traditional SQL backups in a 1.5 hour Max.

I was previously thinking it had to be the performance of my SAN as the repository (NETAPP 2040 with SATA disks). I’ve even been looking into ExaGrid to help alleviate the problem but based on my testing, I’m not fully convinced that we wouldn’t run into the same problem. Unfortunately there are no other logs I can see other than the point it gets to Copying files... which is where the major delay is.

Does anyone have any insight? I opened a Sev 2 case with VEEAM on Tuesday and I’ve yet to speak with anyone…keeps blowing me off saying he’s been delayed/booked.

I figured I’d get more qualified help here anyways.

Thanks,

Brandon

[mkretzer]

Did you Check the performance counters on your repository? How high does the disk latency get when you restore from the repo?

Our restore speed seems to be limited by the repo disk latency and the fact that it seems to be single thread...

Markus


Gesendet von iPad mit Tapatalk

[Brandon0830]

So I mentioned the NetApp 2040 above which is my repository in Production, but during testing, so I could do this during the day I've been using actually using a better SAN that we use for our Dev/Test/Stage environment with 10GB CIFS as the repository off a NETAPP FAS 8020. It still has SATA disks like the 2040's but with Flash Pooling which helps a ton (Disk performance usually about mirrors pretty close to what we use in production which is a FAS3250 with SAS disks).

The 10GB CIFS off the FAS 8020 has been marginally better (instead of 5 hours, it maybe takes 4) for that same 300GB Database. I did run some performance counters on that volume and saw 25ms of latency during the restore along with really bad Read Throughput but I'm not understanding why the throughput is so bad. Like I said above, if it doesn't have to use iSCSI/Staging Server and it just mounts to C:\VeeamFLR on the VEEAM server, the speed is fast.

[tsightler]

There are definitely some known performance issues with SQL restores in v9 and that this is a significant focus for improvement in the next release (9.5). Support may also have access to some hotfixes which provide at least some improvement.

[Brandon0830]

Thanks Tom. Are these performance issues directly related to the iSCSI restore/Staging server do you know? I've been going to crazy trying to figure out whether the problem is VEEAM or my hardware/environment.

[tsightler]

Yes, they are related to that component and there is ongoing work to improve the efficiency of this process.

[Brandon0830]

Just following up on this. I did speak with VEEAM support and he was talking with the "Restore Team" for suggestions. Unfortunately, they did not have any hotfixes available for this issue and really didn't have any suggestions.

However, through a lot of trial and error, I think I have a pretty workable solution.

What I've done is, if I install the VEEAM Console on the SQL server that I'm recovering data to and set that server to the Staging server and select the EXPORT option, then it completely uses FLR and can do point-in-time restores from C:\VeeamFLR. My tests on this have been going very fast so far.

The only downside to this is that the VEEAM console has to be installed on all SQL servers, all restores will have to be done on from SQL server it's going to be restored to, and I'll have to make sure I have I have adequate free space for the Export restore if we don't want to overwrite anything.

-Brandon

[Kellen.Flood]

Veeam Team. Has this issue been resolved in the upcoming V9.5

I would really prefer not to install the console on every SQL server

[foggy]

Veeam B&R v9.5 will introduce certain improvements in the area of SQL restores.

[sandsturm]

Hi

I tested some mssql point in time restores today with Veeam B&R v9.5u2. Backups are stored on a Netapp AllFlash system (A300), the ESXi hosts have mounted the Netapp volumes via NFS over a 10Gb/s Network. The virtual Veeam Backup server uses a local installed MSSQL 2016 as staging server. The proxy and repository role are installed together on a separate virtual server, repository is mounted to the proxy/repository server via 10Gb7s iSCSI network, repository is formatted with ReFS. All Veeam Servers are installed on Windows Server 2016.
I just used a very small database, but the restore took very long. So, in my opinion the performance problem isn't solved with 9.5. I would really be happy, if these restores would be performant, so that I can convice our DBAs to use Veeam as their backup and restore solution instead of the actual one. Does anyone has other or the same experience with mssql restores?

thx and regards
sandsturm

[foggy]

I recommend you contacting support to review the setup and probably identify what can be improved for more optimal dataflow during restore. Thanks.

[kubimike]

Same problem here. Old thread but same issue. Ticket # 05775204 . In V10 same hardware same staging server it would FLY. Im exporting a 200gb DB to the veeam server desktop 40mbps.