Incredibly slow Direc SAN restore

[claudiofolu]

Hello,
We are doing a restore with a Direct SAN proxy and the process rate never goes beyond 8MB/s

The logs shows that the proxy selection is correct, and that the access method is [san]. FibreChannel
The proxy has access to both LUNs, source and target. Different storages.
Any clues?

[SyNtAxx]

I am trying to get direct san restore to even begin to work. I'd be interested in your settings on the luns as viewed from the proxy if you dont mind sharing.

[Vitaliy S.]

@claudiofolu, if you try to restore via NBD or hotadd, what performance do you have? Also have you tried to restore to another LUN?

@SyNtAxx, LUNs should be presented in the same way as you do it when configure direct SAN mode for VM data retrieval.

[claudiofolu]

@claudiofolu, if you try to restore via NBD or hotadd, what performance do you have? Also have you tried to restore to another LUN?

Hello Vitaly, I only have one LUN to test. it´s a 16TB LUN. (VMFS5)
with NBD: 17MB/s

[SyNtAxx]

@claudiofolu, if you try to restore via NBD or hotadd, what performance do you have? Also have you tried to restore to another LUN?

@SyNtAxx, LUNs should be presented in the same way as you do it when configure direct SAN mode for VM data retrieval.

I should have been more clear, but I also didn't want to hijack the thread...

My direct SAN backups work with out the need to present the luns to my proxy server. I suspect his is because I am using the 3par integration properties of Veeam. I have a single lun presented to the proxy server that I want to use for recoveries only. The proxy is a physical machine, with Windows 12012 R2 installed. The test lun in question is vmfs (esxi 5.5) as as viwed from the proxy 'offline' and 'read only'. It seems obvious that it needs to be 'writeable' and presumably 'online' but I've not found any explicit direction in Veeam manuals or online help.

Thanks,

Nick

[Vitaliy S.]

It seems obvious that it needs to be 'writeable' and presumably 'online' but I've not found any explicit direction in Veeam manuals or online help.

I don't believe bringing disks on-line is the requirement. There are no instructions, as setting up the connection for restores is the same as configuring SAN LUNs for data retrieval. Be aware that SAN restores are only possible for thick disks. If it is not the case, then I would suggest looking into restore job log for further information.

My direct SAN backups work with out the need to present the luns to my proxy server. I suspect his is because I am using the 3par integration properties of Veeam.

BTW, you can confirm that your backup jobs can be run in Direct SAN Mode, by unchecking storage integration option. In order to do a quick test like that, just create any dummy VM (even without Guest OS) and try to back it up via direct SAN mode.

Hello Vitaly, I only have one LUN to test. it´s a 16TB LUN. (VMFS5)
with NBD: 17MB/s

What about backup job performance via direct SAN mode? What are the rates and what it the bottleneck in this case? Additionally, do restores via hotadd also have the same performance as NBD mode?

[claudiofolu]

Hello Vitaly,
We´ve opened a support case #00825830
Performance in backup jobs is excellent, 130MB/s
Could the size of the LUN be a problem?
We also checked is the lun was ReadOnly, and no

[Vitaliy S.]

Yes, the performance is indeed almost perfect. Is it a full run? The size of the LUN should not matter, let's see what our support team says after reviewing log files.

[claudiofolu]

Support find that the target, in this case the Datastore we are using to restore, is at 99%
This is strage, since the Storage we are using to restore (EMC VNX 5300) is not running anything else, and the host is physical with 32GB of ram

How can we find the real problem here?

[Vitaliy S.]

What rates do you have for hotadd restores? Is it comparable to network and direct SAN modes?

[claudiofolu]

Didn`t test it. We don`t have a virtual proxy on the DRP site.

[SyNtAxx]

I have gotten my direct san recoveries working. My question now is what rate to I use as the benchmark speed? On the Stats tab of the restore, im seeing 334MB/sec. On the Logs tab, the speed is indicated as 82MB/sec on the last vmdk being recovered. Its a bit confusing.

-Nick

[Vitaliy S.]

Nick, can you please share what you've done to increase the performance of the restore job? Did you restore multiple vmdks or the VM had more than 1 disk?

[SyNtAxx]

The vm I am restoring had multiple vmdk files. One is 60gb and the other is 1.5tb. The SAN I am recovering to have 1400 drives on the disk tier I am using. Ive done nothing but present a test lun and make it writable from the proxies pint of view.

I will say, my test restore has not made any progress in hours. I am wondering the job is hung.

-Nick

[SyNtAxx]

I cancelled my test restore and restarted it. It hangs at the same place. 32% and the metters no longer count down. Time is still, data restored is still, etc. I think I'll have to open a case.

-Nick

[Vitaliy S.]

Yes, please and let us know your case ID. Thanks!

[SyNtAxx]

Case # 00828648

[Delo123]

I can confirm the san volume needs to put online by hand, and write access needs to be setup on san side. Also for direct SAN Backups disks need to be put online by hand (but not writable of course)
However we also cannot find a Reason for slow restore speeds. SAN is able to do 2GB/s+ and Backup repository is also able to at least read at a few hundred mb/s (72 7K4 SATA Disks in multiple Raid 5s striped in Windows). However we never get over 22MB/s, NDB restores are about the same. We could never find a reason Network is 10Gbit, SAN is FC connected.

[Vitaliy S.]

Guido , volumes should not be online, at least in my lab I have backup job running in a direct SAN mode with offline volumes. Let our support team look through your restore job logs, they might be able to find the reason for this slow performance.

[WimVD]

We are also using 3PAR integration. Is direct SAN restore possible in this configuration?
I actually doubt it since LUN's are only presented at the moment of backup and even then only snapshots not the original LUN...
Is Veeam smart enough to present the original LUN for a restore with 3PAR integration?

In my testing I only get NBD restores...

[foggy]

If you have the LUN itself presented to the proxy server in the read-write mode, direct SAN restore will be available.

[WimVD]

Ok, will test with presenting the lun...
Other question, the documentation you linked to mentions the following:
The Direct SAN Access transport mode can be used to restore VMs with thick disks only

Do they need to be eager zeroed or is lazy sufficient?

[WimVD]

Ok, just tested with presenting the LUN in read/write and can confirm this makes SAN restore available.
However the disk needs to be online for it to work. If the disk is offline it will attempt SAN restore but will failover to network:
Unable to execute write operation using advanced transport, failing over to network.

Not sure if this is a bug or expected behaviour but keeping my production lun's online makes me a little nervous...
Would be nice if we could keep them offline. Even better would be if Veeam could use the 3PAR storage integration to selectively present the production LUN when needed for a restore...

Direct SAN restore performance was a little disappointing.
80MB/s is not bad but it's the same as restoring over network.
Was hoping for an increase but I guess my bottleneck is somewhere else...

[Vitaliy S.]

Not sure if this is a bug or expected behaviour but keeping my production lun's online makes me a little nervous...

There is no need to be nervous since Veeam B&R server automatically sets SAN policy to offline mode and automount to disabled for all new volumes on the proxy server.

[Gostev]

However the disk needs to be online for it to work.

This should not be the case?

[Delo123]

I can also confirm disks need to be put online or job will fallback to ndb
@Vitaly we already had veeam support looking at this, they confirmed we need to put disks online first (not initialize of course!), but reason for slow restores could not be found.
We also changed a reg entry for local data movers but it didn't help much. We could have done further investigations but as a "one man show" i cannot spends week doing trail & error so we learned to live with it for now.

[foggy]

Do they need to be eager zeroed or is lazy sufficient?

Both thick disks types are supported.

[claudiofolu]

Hello everybody
I want to give you an update on this.
The problema was that some read cache on the VNX box was disabled due to a battery failure. After re-enable it, restore rate was around 100MB/s.

Regards,

[SyNtAxx]

Ive been doing additional testing. My actual SAN restore rate is about 90MB/SEC. I have a problem though, large vm restores fail. I have many vms over 1TB and if I cant recover in the fastest way possible it is problematic. Anybody else have failed direct SAN restores on large vms?

-SyNtAxx

[WimVD]

We have had issues with large disk restores (VM File) over network failing with "NFC server is busy"
This was the whole reason we are looking at direct SAN restores. Haven't tested large SAN restores yet so hopefully they are more stable.