SAN Restore Performance

[BackupTest90]

Hey Guys,
I would like to ask you about your SAN-Restore performance?
We have two different Storage Array (3PAR and Nimble AF) and got only a restoreperformance from aprox. 50MB/s to 80MB/s by an entire VM-restore. It is no difference, if it a VM with one or multiple VMDKs (restore mode thick eager). Both arrays are connected with multiple 8Gbit/s FC connections and the HW-Proxys have per fabric one 8Gbit/s connection.

Do you have other values? What are your experience? In my opinion, it is very slow. Is there any possiblity to speed up this?

Regards,

[mdiver]

Hi BackupTest90.

What's you speed with an active-full from the same box? I would expect almost the same rates for a full-VM recovery through the same proxy. Especially for AF.

Are you sure you're getting a SAN based restore? Does it say so in the protocol?

You can only do SAN based restores with thick-provisioned disks.
Also you need write access via your proxy accessing the LUNs.
In the other cases Veeam will failover to network silently (if allowed to).

Thanks,
Mike

[BackupTest90]

Hi Mike,
yes, the protocol said e.g "Restoring Hard disk 4 (90 GB) : 58,9 GB restored at 70 MB/s [san]" and "SAN" means, all prerequesites fullfills (Thick Disk, Proxy Access to VMDKs,..)

I've did an active full of two VMs and the result below:

VM1
29.07.2020 19:32:17 :: Using backup proxy XXXX for retrieving Hard disk 1 data from storage snapshot
29.07.2020 19:32:36 :: Hard disk 1 (60 GB) 38,9 GB read at 614 MB/s [CBT]

VM2
29.07.2020 19:32:19 :: Using backup proxy XXXX for retrieving Hard disk 1 data from storage snapshot
29.07.2020 19:32:36 :: Hard disk 1 (60 GB) 44,5 GB read at 513 MB/s [CBT]

I thought that I got similar restoretimes..

Regards,

[Andreas Neufert]

DirectSAN restore is way slower than the backup as any write has to be coordinated through vCenter. So vcenter connections and performance is as well critical here.

I suggest to add a HotAdd proxy for restore which should give you full throttle to what the storage system support from throughput perspective.

To reduce overhead on vcenter. You can try to add an esxi host by IP address to Veeam as managed server. Then use this one to restore the VM to. This allowes the ESXi host directly to control the reservations for writing blogs and is ususally faster for Direct SAN restore.

Overall the DirectSAN restore protocol is by design of VMware not the fastest one and potentially even NBD-network mode is faster.

[PetrM]

Hello,

One more idea is to ask our support team to check logs and to look for the "bottleneck", maybe restore speed is decreased due to slow data read from repository and not because of transport mode.

However, as a first step of troubleshooting it makes sense to perform tests suggested by Andreas.

Thanks!

[BackupTest90]

Oh, we identified a problem, with the MPIO Multipathing and solve this.

These are our SANvalues now

Restoring Hard disk 1 (25 GB) : 7 GB restored at 227 MB/s [san]
Restoring Hard disk 5 (200 GB) : 199 GB restored at 113 MB/s [san]
Restoring Hard disk 2 (100 GB) : 100 GB restored at 145 MB/s [san]
Restoring Hard disk 4 (100 GB) : 99,9 GB restored at 96 MB/s [san]
Restoring Hard disk 3 (100 GB) : 99,8 GB restored at 134 MB/s [san]


The Hotaddvalues below:(Same VMs as above, THIN Restore)
Restoring Hard disk 4 (25 GB) : 7 GB restored at 204 MB/s [hotadd]
Restoring Hard disk 2 (100 GB) : 100 GB restored at 140 MB/s [hotadd]
Restoring Hard disk 3 (100 GB) : 99,8 GB restored at 123 MB/s [hotadd]
Restoring Hard disk 1 (100 GB) : 99,9 GB restored at 90 MB/s [hotadd]
Restoring Hard disk 5 (200 GB) : 199 GB restored at 98 MB/s [hotadd]

Thanks for you ideas, I will test this with a standalone ESX.

[lvmusso]

Hi Martin, could you please clarify when you said "Oh, we identified a problem, with the MPIO Multipathing and solve this." ... cause we're facing the same speed problem, and mpio could be an issue.

Thx!

[Serger]

Oh, we identified a problem, with the MPIO Multipathing and solve this.

Hello Colleague!
Your restore speed is lookong good for now! Even better then "hotadd"
Can you tell please, are you using a Nimble storage for Backup repository? And what kind of multipathing issue you got?

Becouse we are having a similar issue with restore in SAN mode on 3Par from StoreOnce Catalyst store. We got two identical infrastructures and in one of them we can reach restore speed about 100-200 MB/s per VMDK. But in another one it is about 50-80 MB/s per disk. All hardware and software is the same. But, maybe we have lost some tuning options...

Thank you!

[PetrM]

Hi Sergey,

I'd suggest to raise a support case as well so that our engineers can examine this performance issue a bit deeper. Perhaps, the problem is not related to the transport mode in which proxy writes data to the target datastore but occurs due to slow read from the backup repository or because of slow data transmission from the repository to the target proxy. Please don't forget to share a support case ID with us.

Thanks!

[foggy]

It is also worth testing exactly the same restore operation, say full VM restore of the same VM being backed up to two different targets. That way you'll get an apples-to-apples restore performance comparison as the actual data workload does matter.