Incredibly slow Direc SAN restore

[foggy]

You can check whether other restore modes work better for you. I would say that we're aware of the issue and currently monitoring the demand for implementing eager zeroed disks restore.

[WimVD]

+1 for implementing eager zeroed disk restores

And since we're feature requesting it would be nice to use storage integration (3par, netapp,...) to present the destination restore disk to the backup server 'on demand' when initiating a restore.
Now we have to manually present it first and bring it online when doing direct san restore...

[SyNtAxx]

You can check whether other restore modes work better for you. I would say that we're aware of the issue and currently monitoring the demand for implementing eager zeroed disks restore.

Maybe i'm off base here, but if you have to test the winds to see if anybody 'wants/demands SAN restores to work as expected i think that is a problem. I resent the fact I have to take a snapshot of a lun I backed up via SAN and mount it to a vm as an RDM in order to restore Gigantic vms via hot add in order to get decent restore speeds. If its not obvious, I vote for SAN restores to work properly.


-Nick

[chjones]

The case I opened is still open with Veeam and has kind of been taken over by VMware in their own case, so we are waiting on them. The last update I received was last Wednesday and the Support person said he was in discussion with the engineering team and haven't come to any conclusion as to the cause of slow restore performance with Lazy Zeroed Disks and SAN Restore Mode.

I had been told earlier that they were able to reproduce the issues, so that's something I guess.

Just came across the same issue with my Direct SAN restore testing. Is there any update on your case?

[foggy]

Maybe i'm off base here, but if you have to test the winds to see if anybody 'wants/demands SAN restores to work as expected i think that is a problem. I resent the fact I have to take a snapshot of a lun I backed up via SAN and mount it to a vm as an RDM in order to restore Gigantic vms via hot add in order to get decent restore speeds. If its not obvious, I vote for SAN restores to work properly.

Nick, I cannot recall the initial reasons for the current implementation, probably the need for disk preallocation that takes quite long for large eager zeroed disks and can seriously affect job duration (while data transfer for lazy zeroed disks starts right away). However we did not have much complaints regarding that until recently, that's why I was talking about demand. Anyway, we're performing some research in this regard. Thanks again for your feedback.

[doggatas]

Hi

Just wondering if there is anyway to use network mode as opposed to SAN mode for restores only? or is the mode dependant on which mode is set for the backup proxy used? What if I only have/need one backup proxy, i.e. the Veeam server is a physical server? Can we force restore to be completed of network mode?

Regards,
David

[Vitaliy S.]

Hi David,

You're correct that the restore mode depends on the proxy configuration. If you install a physical proxy server and do not mount any LUNs network restore mode will be used all the time.

P.S. in order to restore over the SAN mode, I believe read/write access is required, so if you could configure your proxy server to operate in read-only mode which is sufficient for doing backups, then you will achieve your goal as well.

Thanks!

[b.vanhaastrecht]

Do we still need to put the presented LUN's online in Windows? As we can't restrore over SAN while LUN's are presented read/write, but are Offline (as desired).

[Delo123]

Yes, they need to be online for direct SAN restore to work...

[b.vanhaastrecht]

And is this marked as a bug by Veeam? Why are backups posible, but not restores in Offline mode?

[Delo123]

Not sure, maybe Veeam can answer.
However as you of course know, do not initialize the disks, just put them online

[Vitaliy S.]

Do we still need to put the presented LUN's online in Windows? As we can't restrore over SAN while LUN's are presented read/write, but are Offline (as desired).

Bastiaan, I have double-checked with the QA team and LUNs do not need to be online when using direct SAN restore. If you see another situation, then please contact our support team for investigation.

[Taran]

One of my customer's is encountering this, I've asked them to post the support case number here. Here are my notes:

Checked:

• Proxy with DAS Repository
  Proxies have been configured to use SAN mode
  Restores log indicates SAN mode is being used
  Proxies have 12 cores, but are configured for 20 concurrent tasks
  Backup jobs are using “Local storage” as the storage optimization. (There aren’t any huge 16TB+ backup files)
  Local backup job in Melbourne then backup copy to the repository in Sydney.
  Restores are from the repository in Sydney to the Sydney DataStore, the vCenter is in Melbourne
  Melbourne is production and the network is far busier.
  Backup speeds are 125+ MB/s
  Full VM restores, the plan is to use these restored VMs for replica mapping.
  Eventual plan is to use the backup copy job as a source for the replication jobs.
  Quick view of Veeam One reports for the datastores did not yield any issues on the datastore side, and alarms in the infrastructure didn’t seem related.
  Restore jobs are selected use the same disk type
  Disks are Thick provisioned with mostly Lazy Zeroed Disks, a few eager zeroed disks, Restored disks are the same type.

Tests:
All tests were done with Lazy Zeroed Disk VMs only

• Restore in Sydney with the current setup: speed 26MB/s
  Change the Sydney proxy to network mode: speed 76MB/s
  ***Network mode speed over a 10GB, non-management interface was 3 times faster to do the same restore***
  Try to restore a VM from a regular backup job, instead of a backup copy job: speed 75MB/s
  Restore in Melbourne in SAN mode: 60MB/s
  Restore in Melbourne in network mode: 35MB/s

Problem:
Restores are functioning, however the speed is not as expected.

[MrDave]

Hi my case is what Taran is talking about. We are experiencing extremely slow restores as well. Case number 01006622.

We have also created a VM with Thick Provisioned Eager Zero and tried to restore as the same type with the same slow results ~25MB/s.

Our SAN is a Dell Compellent with dual SC8000 controllers.

Hope someone can help. Thanks

[Taran]

The customer is having difficulty posting, so I thought I'd pop the case number in: 01006622

[PG5150]

I was wondering if there was any feedback in relation to the post from Taran and MrDave.

[SyNtAxx]

I was wondering if their is a solution to the problem yet.

[Gostev]

I overheard recently that support now has a patch that switches thick disk restore mode to eager zeroed.

[SyNtAxx]

I overheard recently that support now has a patch that switches thick disk restore mode to eager zeroed.

Fantastic! is it general release for public consumption, or do you have to ask nicely for it ?

Thanks.

[Gostev]

With rare exceptions, you don't have to be nice to get hot fixes which are already in support's hands

[Taran]

Here's an update for everyone. The hotfix supplied by support improved the restore speeds to 200MB/s. There is some further work happening, however this is a significant improvement.

Asking support nicely is the way to go to get the hotfix (being nice never hurts). It might be a good idea to reference this forum post.

[MrDave]

Worked a treat. We do believe however that we should be able to achieve speeds faster than 200MB/s as we have some SSD trays in our SAN. There is further investigations happening there.

[Lasr]

Hi,

is this hotifx included in B&R 8.0 Update 3?


Regards
Lasr

[foggy]

No, chances are this will be addressed in v9.

[Gostev]

Based on hot fix testing, for v9 we are simply planning to make this a default behavior.
Right now, there is simply not enough time to build and test delivering this as an option.

[SyNtAxx]

What is the path/hotfix number so I may request it?

-Nick

[veremin]

Within an existing support ticket just refer to the Taran's post. I think it should be enough to get the said fix. Thanks.

[Anders]

With a EMC VNX5300 attached with 8G FC I got these numbers
The VM I've testet this with is 500 GB in size. (1x 100 GB, 2x 200 GB VMDKs)
Direct SAN backups run at around 150 MB/s
Network backup runs at 70 MB/s (1G Network)
Restore via Network runs at about 70 MB/s (1G Network)
Restore via DirectSAN without patch runs at 25-45 MB/s
Restore via DirectSAN with the patch runs at about 99 MB/s
The problem with the patch is that it takes ages before the restore starts because it creates an empty zeroed VMDK file first, and that took longer than the entire VM Restore does via Network. We are pretty sure this is because we have VAAI disabled due to a bug in the EMC VNX firmware.

In the end we uninstalled the patch.

So making the patch the default in v9, without a way to disable this, will make people with VAAI disabled cry!

Also, remember to include your Veeam build number when requesting the patch via support - will speed up getting the patch by a day or two!

[foggy]

There will be a registry value to switch the behavior. Btw, thanks for this feedback, much appreciated.

[m1kkel]

Hello.

Just finishing setting up restore and backup via san mode. Had some issues with restore, because of lun's were write protected on my proxy server, which is windows server 2012R2, because of default san policy.

Anyway, the restore of an entire VM with only 1 restore point, was slow, around 77 MB / sec. Tried Hotadd, which was faster, around 90 MB / sec.

Here is the really weird part. Server is 2012R2 running storage spaces, 12 disk presented as jbod without raid controller, SATA 7,2K Mirror with deduplication. CPU is xeon something with 4 cores, and 32 GB memory. Now, i am copying a file internally on the server, from the V: drive (for veeam ) to the C drive, which is a SSD disk (in mirror on another controller)
Copying a VBK file from the backupupjob folder called "DirectSan_test_Hostingtest" to the C: drive, i get a speed rating around 30-130 / sec. (size is around 30GB) Copying some VIB files from other folders around 17 GB in size, i get speed around 500-700 MB pr. second which is as expected. And some of the other VIB files from some of the other backup jobs are REALLY slow just copying internally from V: to C: and some are extremely fast.
Volume is NTFS, formatted in 64 KB cluster size, and formatted with the option for large drives.

This does not make any sense to me, is something fragmented, and maybe some of the chunks are placed randomly on the volume, and maybe some are aligned up nicely?

Settings in veeam for storage is local target, and optimal compression.
In veeam, i am not using the option for "Align backup file datablocks"

Suggestions?