VBR11 : Extremely slow recovery

[ThierryF]

Hello,

I am performing VM Recovery of my SCCM SiteServer, just acting as database server,
SCCM Source_CM File Library being located on our file server. So just database and SCCM binaries.
Not a lot of small files like a File Server could serve. VM has 1.4TB vmware datastore space, powered
by IBM FLASYSTEM NVME Array, on 16Gbit SAN Switches. As Network, 10Gbit lan...

During normal operation, no perf issue at all on our infrastructure, nor during backups, nor
during normal workload. Backups fly, no IO Prob on NVME backend.

Today, SCCM SS got corrupted and I have been asked for performing VM Recovery.
Used recover point composed of 1 FULL followed by 4 INCR backups.

Looking at Veeam backup repository, I only see 5-10MB/Sec IOs by VeeamAgent.exe
to VM's VIB and VBK files. No other jobs running on infra nor tape copy that could cause
or explain IO Load on backup repository.

It is increadible slow recoverying my VM. Started 6 HOURS ago, according recovery stats it only recovered
450GBs on 1.4TB VM Size (I estimate 1.2TB Data usage in VMDKs on the 1.4 Total disk capacity)
Stats predict another 14 hours for completing recovery.

Disks being recovered are disks storing TRLOG, IDX, MDB Files ... So, big files.
Between Veeam infra (Server, 2 proxies and 1 File Server and VCenter 7.0.3 infra), 10Gbit Network.

How is it possible VBR to be so slow recovering while backups run without issue ?
Perf issue on File Store ? I doubt : I have daily tape copies that copy last backups to tapes
that runs without problem at 200-250MB/Sec.

Issue merging VIBs/VBK to retrieve blocks to send back to Vmware for recovery ?

Already performed big VMs recoveries (but not using VBR11a (+last security fix) yet)
6 hours for 450GB ... seriously ... Avg 20-22MB/Sec with such a config .... I am clueless...
It already recovered 5 VHDs and 2 VHDs are still recoverying, one of 1TB, one of 200GB.

What could be the issue ?

Thanks

Th

[ThierryF]

Job log :
---- edited by moderator
16/03/2023 13:17:35 Using proxy 01 for restoring disk Hard disk 7
16/03/2023 13:17:35 Using proxy 02 for restoring disk Hard disk 6
16/03/2023 20:02:30 Restoring Hard disk 7 (1024 GB) : 103,4 GB restored at 4 MB/s [nbd] # RECOVER STILL IN PROGRESS
16/03/2023 15:40:18 Restoring Hard disk 6 (100 GB) : 33 GB restored at 4 MB/s [nbd]

[ThierryF]

@Moderator : Had anonimyzed joblog , disks #7 and #2 still recovering.
Th

[PetrM]

Hello!

I shortened your second post for a better readability. Please don't post log snippets on this forum, the debug logs analysis must be performed by our support engineers.

There are many reasons why restore can be slower than backup, for example: backup is running in SAN mode and the restore proxy works in Network mode (btw from the log I see it's NBD), slow read from the backup repository (f.g. storage level deduplication), decompression is slow due to CPU load on the backup repository and many others.

Anyway, instead of trying to guess on this forum what it might be, I recommend to contact our support team. What you describe looks like a typical technical issue. I suggest checking that the same proxy servers that are used during backup are used for restore as well, try a virtual proxy and restore in HotAdd mode instead of Network, open a support case for a closer look at the debug logs. Also, you may share a support case ID with us.

Thanks!

[ThierryF]

Hello,
Thanks for inputs. You pointed same I did.
Backup and restore are Network mode, with 10Gbit lan accross sites.
About CPU load, I checked, 1 recover stream on proxy1 and one on Fileserver. Server CPU Load is 1%.
Looking at perf mon, I see 1-5MB/Sec IO Read by veeamagent on File Repository side.

During recover I had jobs that started and completed at 80-100MB/Sec Write ...

FileRepoisotry is ReFS Volume, 6 HDDs Raidset with read/write cache, that reads at 150MB/Sec when tape copies.
No other IOs to repository.

Just think about decompression issue or merging VIBs/VBK for recovey ....
Posted to forum as I try to push up recovery asap, stats show still 14hours to go .... but I will open case tomorrow...
Thanks

[ThierryF]

Case Number is 05950211

[ThierryF]

Recovery time difference recoverying SAME VM recover point using SAME infrastructure
(Veeam Proxy, Veeam File repoistory, 10Gb Lan, Networking Swithces, Forti Firewall, Cisco Hardware powering ESX ....):

to 7.0U3 :
--> Start 16/03/2023 13:15:22
--> End 19/03/2023 06:15:47
--> Duration 65:00:25 ######
--> Speed 6,27 MB/Sec ######

To 6.7 :
--> Start 17/03/2023 21:49:40
--> End 8/03/2023 06:39:04
--> Duration 08:49:24 ######
--> Speed 46,22 MB/Sec ######

[Gostev]

try a virtual proxy and restore in HotAdd mode instead of Network

This is the way (c) will likely make the restore go up to 100x faster.

[ThierryF]

Need to change the way to take backups, skipping CBT, to avoid Veeam to fix the problem ?

[ThierryF]

Virtual proxy and HotAdd allow faster recovery, maybe, but if so, why does Veeam still continue to offer NBD feature ?
Present me any sysadmin or customer to setup an infrastructure for 5MB/Sec backup recovery speed using 10Gb network infra ???

On top of, assuming NBD is slower, ok, ugrading Veeam 10->11A and Vmware ESX 6.7->7.0U3 shoud not make recovery more than 8 times slower ...

Please may I invite you to read Veeam doc : https://www.veeam.com/blog/vmware-backu ... ation.html :

Hot Add transport mode, The Bad & Ugly :
. Proxies consume resources on vSphere cluster
. The hot add process is slow to start due to changes to configuration to run
. Disables CBT on backup proxy server. (Read as more IO to Source Data Store and load on Storage infra)
. #1 source of stuck (Consolidation Needed) and hidden snapshots.

Network transport mode :
The Good :
. Enables 100% virtual installation and deployment
. Can be quite fast on 10Gb Ethernet networks!
The Bad & Ugly :
. Normally 10-20 MB/s at best for backups and restores.

Just getting surprised about mixing "Can be quite fast on 10Gb Ethernet networks!" and "Normally 10-20 MB/s at best for backups and restores." ???
With Vmware 6.7, restore at 46MB/Sec, with vmare 7.0, 6MB/Sec is a normal speed ? Seriously ...

[Gostev]

There's no "skipping CBT" involved when changing a backup mode.

We keep offering NBD feature because it is good enough at the edge (ROBO environments with a few tiny VMs).

As you can see, the "Veeam doc" you're inviting me to read is from 2016. Obviously it does not automatically apply to any random point in time in future, including in the future where VMware broke NBD performance. By the way, this blog is essentially a copy/paste of my presentation at VMworld 2015

[ThierryF]

For the community ...

Using VBR 11a, NBD-mode 1.4TB SCCM VM recover completed in 65 hours.
Adding Vmware Proxy and recoverying same point in HotAdd mode completed in ... 1h 40 ...
Nearly 40 Times faster using SAME infrastructure ...

You will definitively have to look at/review your envs if still using NBD recoveries.
I opted for RedHat 9 Linux VM, 16GB Ram and 1 VCPU per allowed stream (6 in my case).
Proxy-VM need to be powered by ESX node that has acces to all used datastores or if,
like us, you have restricted ESX to some datastore, so, additional proxies will be needed.

BTW, on our test env (VBR 12 and ESX 8 ), extremely poor NBD perf is still present ...

Th