Slow File Level Restore after upgrade to Veeam 11

[JRBeaver]

I have an open case with Veeam support, case ID: #02303616
I'm wondering if anyone else has seen extremely slow FLR mounts after upgrading from Veeam 10 to 11.

Typically jobs that would take from 1 to 3 hours to mount are now taking 6 to 9 hours to mount, in one some cases the backups do not mount at all after 28 hours. Others have mounted after 20 and 27 hours.

This is only happening with one of our clients but pretty consistently for them. They are running Windows 2012 R2 servers mostly with a VM as the backup server, this VM is a second DC in the site running on a dedicated backup host for replication. Most sites have QNAP NASs of various sizes although the main site has a Thecus NAS running Windows storage server.

The similarity with all the slow restores is large VMs from 4 to 12 TB, all VMs 2 TB and less don't appear to have the issue.

It has been mentioned that fragmentation could cause the slow restores, and I don't doubt it could, however the successful restores prior to the V 11 upgrade were only days or weeks prior. I am also running a defrag and compact on the critical server we are trying to restore from, however this is likely to take until the weekend or after at 18% after 20 hours.

Just clutching at straws here to see if anyone else has observed this, and ideally found a resolution.

[HannesK]

Hello,
do you have the performance issues _only_ with restore, or also with backup? As far as I see, you are already testing reg keys that bring back the V10 behavior.

In general: yes, active full or defragment files reduces mount times.

Best regards,
Hannes

[JRBeaver]

Hi Hannes,
Thanks for the reply. I hadn't noticed any slow down in the backups, I've just checked some history and they don't seem to have increased.

I've had one restore mount in 3 hours 14 minutes after the reg key was added, still slower than the average 1:45 for that site, I'm testing with some more sites also. However one of the original sites that has had the changes is still trying to mount after 4 and a half hours, previous were 1:07 and 1:49.

[HannesK]

hmm okay... did you / support check that definitely the correct mount-server is used? just to make sure that it doesn't happen just by chance that you are mounting over WAN.

[JRBeaver]

yes, the mount server is the backup server and we also tried using the backup host.
FYI, the environment for this site is production host with production VMs, backup host which hosts the backup VM, all equipment in the same rack including the NAS and all connected to the one switch.
I also built separate VMs for Veeam on both hosts and imported the backups into these with the same result.

[HannesK]

okay, I expected when you talk about clients, that it is about a distributed environment with slow network connections in between.

As of now, I can only recommend to continue working with support to find the reason. Issues with restores while having no issues with backups are nothing I have seen so far for V11.

[JRBeaver]

Thanks HannesK, support do think it's a known issue, just that the fix they have isn't working in this case.

As for clients, no it's not over slow connections, we're an MSP so we provide Veeam backups to all our virtualised clients, this one has multiple sites and each has their own backup server with all local infrastructure, the only WAN involved is for the offsite copies.

[desmond shio]

Hi JRBeaver,

I also facing same issues, any update so far?

Thanks

[HannesK]

as far as I see in the case comments, UseUnbufferedAccess = 0 solved it in this situation. But that doesn't mean that it also solves it for you.

[pirx]

Case 05084535: our first FLR session is currently running after v10 -> v11a. Restore of 18 files with 37,9GB is now already running for more than 17h.

[foggy]

Thanks for opening a case. I believe you've already checked that the correct mount server is being utilized, right? What about backup performance? Have you run any backup jobs since the upgrade and what was the performance?

[pirx]

It's one of our physical Veeam proxies which was used for the last 5 years. Backup performance to XFS repository is not different to before upgrade.

Code: Select all

23:39:17	Using backup proxy xxxx for retrieving Hard disk 1 data from storage snapshot on vdesvc01 	00:07
23:39:25	Hard disk 1 (50 GB) 338 MB read at 131 MB/s [CBT]	00:08
23:39:42	Using backup proxy xxxx for retrieving Hard disk 2 data from storage snapshot on vdesvc01 	02:15
23:41:59	Hard disk 2 (30 GB) 512 MB read at 146 MB/s [CBT]	00:07
23:42:09	Finalizing 	00:18

[foggy]

The restore session logs should point to the bottleneck, let's give our engineers time to review those.

[pirx]

Sorry for the noise, our case is firewall related!