Host-based backup of VMware vSphere VMs.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Incredible slow Direc SAN restore

Post by foggy »

You can check whether other restore modes work better for you. I would say that we're aware of the issue and currently monitoring the demand for implementing eager zeroed disks restore.
WimVD
Service Provider
Posts: 60
Liked: 19 times
Joined: Dec 23, 2014 4:04 pm
Contact:

Re: Incredible slow Direc SAN restore

Post by WimVD »

+1 for implementing eager zeroed disk restores :)

And since we're feature requesting it would be nice to use storage integration (3par, netapp,...) to present the destination restore disk to the backup server 'on demand' when initiating a restore.
Now we have to manually present it first and bring it online when doing direct san restore...
SyNtAxx
Expert
Posts: 149
Liked: 15 times
Joined: Jan 02, 2015 7:12 pm
Contact:

Re: Incredible slow Direc SAN restore

Post by SyNtAxx »

foggy wrote:You can check whether other restore modes work better for you. I would say that we're aware of the issue and currently monitoring the demand for implementing eager zeroed disks restore.

Maybe i'm off base here, but if you have to test the winds to see if anybody 'wants/demands SAN restores to work as expected i think that is a problem. I resent the fact I have to take a snapshot of a lun I backed up via SAN and mount it to a vm as an RDM in order to restore Gigantic vms via hot add in order to get decent restore speeds. If its not obvious, I vote for SAN restores to work properly.


-Nick
chjones
Expert
Posts: 117
Liked: 31 times
Joined: Oct 30, 2012 7:53 pm
Full Name: Chris Jones
Contact:

Re: Incredible slow Direc SAN restore

Post by chjones »

The case I opened is still open with Veeam and has kind of been taken over by VMware in their own case, so we are waiting on them. The last update I received was last Wednesday and the Support person said he was in discussion with the engineering team and haven't come to any conclusion as to the cause of slow restore performance with Lazy Zeroed Disks and SAN Restore Mode.

I had been told earlier that they were able to reproduce the issues, so that's something I guess.
dmitri-va wrote: Just came across the same issue with my Direct SAN restore testing. Is there any update on your case?
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Incredible slow Direc SAN restore

Post by foggy »

SyNtAxx wrote:Maybe i'm off base here, but if you have to test the winds to see if anybody 'wants/demands SAN restores to work as expected i think that is a problem. I resent the fact I have to take a snapshot of a lun I backed up via SAN and mount it to a vm as an RDM in order to restore Gigantic vms via hot add in order to get decent restore speeds. If its not obvious, I vote for SAN restores to work properly.
Nick, I cannot recall the initial reasons for the current implementation, probably the need for disk preallocation that takes quite long for large eager zeroed disks and can seriously affect job duration (while data transfer for lazy zeroed disks starts right away). However we did not have much complaints regarding that until recently, that's why I was talking about demand. Anyway, we're performing some research in this regard. Thanks again for your feedback.
doggatas
Enthusiast
Posts: 38
Liked: 2 times
Joined: Jul 24, 2012 1:15 am
Full Name: David O
Contact:

Re: Incredibly slow Direc SAN restore

Post by doggatas »

Hi

Just wondering if there is anyway to use network mode as opposed to SAN mode for restores only? or is the mode dependant on which mode is set for the backup proxy used? What if I only have/need one backup proxy, i.e. the Veeam server is a physical server? Can we force restore to be completed of network mode?

Regards,
David
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Incredibly slow Direc SAN restore

Post by Vitaliy S. »

Hi David,

You're correct that the restore mode depends on the proxy configuration. If you install a physical proxy server and do not mount any LUNs network restore mode will be used all the time.

P.S. in order to restore over the SAN mode, I believe read/write access is required, so if you could configure your proxy server to operate in read-only mode which is sufficient for doing backups, then you will achieve your goal as well.

Thanks!
b.vanhaastrecht
Service Provider
Posts: 833
Liked: 154 times
Joined: Aug 26, 2013 7:46 am
Full Name: Bastiaan van Haastrecht
Location: The Netherlands
Contact:

Re: Incredibly slow Direc SAN restore

Post by b.vanhaastrecht »

Do we still need to put the presented LUN's online in Windows? As we can't restrore over SAN while LUN's are presented read/write, but are Offline (as desired).
======================================================
Veeam ProPartner, Service Provider and a proud Veeam Legend
Delo123
Veteran
Posts: 361
Liked: 109 times
Joined: Dec 28, 2012 5:20 pm
Full Name: Guido Meijers
Contact:

Re: Incredibly slow Direc SAN restore

Post by Delo123 »

Yes, they need to be online for direct SAN restore to work...
b.vanhaastrecht
Service Provider
Posts: 833
Liked: 154 times
Joined: Aug 26, 2013 7:46 am
Full Name: Bastiaan van Haastrecht
Location: The Netherlands
Contact:

Re: Incredibly slow Direc SAN restore

Post by b.vanhaastrecht »

And is this marked as a bug by Veeam? Why are backups posible, but not restores in Offline mode?
======================================================
Veeam ProPartner, Service Provider and a proud Veeam Legend
Delo123
Veteran
Posts: 361
Liked: 109 times
Joined: Dec 28, 2012 5:20 pm
Full Name: Guido Meijers
Contact:

Re: Incredibly slow Direc SAN restore

Post by Delo123 »

Not sure, maybe Veeam can answer.
However as you of course know, do not initialize the disks, just put them online :)
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Incredibly slow Direc SAN restore

Post by Vitaliy S. »

b.vanhaastrecht wrote:Do we still need to put the presented LUN's online in Windows? As we can't restrore over SAN while LUN's are presented read/write, but are Offline (as desired).
Bastiaan, I have double-checked with the QA team and LUNs do not need to be online when using direct SAN restore. If you see another situation, then please contact our support team for investigation.
Taran
Veeam Software
Posts: 87
Liked: 10 times
Joined: Jul 16, 2013 5:05 am
Full Name: Taran
Contact:

Re: Incredibly slow Direc SAN restore

Post by Taran »

One of my customer's is encountering this, I've asked them to post the support case number here. Here are my notes:

Checked:
  • Proxy with DAS Repository
    Proxies have been configured to use SAN mode
    Restores log indicates SAN mode is being used
    Proxies have 12 cores, but are configured for 20 concurrent tasks
    Backup jobs are using “Local storage” as the storage optimization. (There aren’t any huge 16TB+ backup files)
    Local backup job in Melbourne then backup copy to the repository in Sydney.
    Restores are from the repository in Sydney to the Sydney DataStore, the vCenter is in Melbourne
    Melbourne is production and the network is far busier.
    Backup speeds are 125+ MB/s
    Full VM restores, the plan is to use these restored VMs for replica mapping.
    Eventual plan is to use the backup copy job as a source for the replication jobs.
    Quick view of Veeam One reports for the datastores did not yield any issues on the datastore side, and alarms in the infrastructure didn’t seem related.
    Restore jobs are selected use the same disk type
    Disks are Thick provisioned with mostly Lazy Zeroed Disks, a few eager zeroed disks, Restored disks are the same type.
Tests:
All tests were done with Lazy Zeroed Disk VMs only
  • Restore in Sydney with the current setup: speed 26MB/s
    Change the Sydney proxy to network mode: speed 76MB/s
    ***Network mode speed over a 10GB, non-management interface was 3 times faster to do the same restore***
    Try to restore a VM from a regular backup job, instead of a backup copy job: speed 75MB/s
    Restore in Melbourne in SAN mode: 60MB/s
    Restore in Melbourne in network mode: 35MB/s
Problem:
Restores are functioning, however the speed is not as expected.
MrDave
Novice
Posts: 3
Liked: never
Joined: Sep 02, 2015 6:50 am
Contact:

Re: Incredibly slow Direc SAN restore

Post by MrDave »

Hi my case is what Taran is talking about. We are experiencing extremely slow restores as well. Case number 01006622.

We have also created a VM with Thick Provisioned Eager Zero and tried to restore as the same type with the same slow results ~25MB/s.

Our SAN is a Dell Compellent with dual SC8000 controllers.

Hope someone can help. Thanks
Taran
Veeam Software
Posts: 87
Liked: 10 times
Joined: Jul 16, 2013 5:05 am
Full Name: Taran
Contact:

Re: Incredibly slow Direc SAN restore

Post by Taran »

The customer is having difficulty posting, so I thought I'd pop the case number in: 01006622
PG5150
Lurker
Posts: 1
Liked: never
Joined: Jul 16, 2014 5:26 am
Full Name: Philip Gardiakos
Contact:

Re: Incredibly slow Direc SAN restore

Post by PG5150 »

I was wondering if there was any feedback in relation to the post from Taran and MrDave.
SyNtAxx
Expert
Posts: 149
Liked: 15 times
Joined: Jan 02, 2015 7:12 pm
Contact:

Re: Incredibly slow Direc SAN restore

Post by SyNtAxx »

I was wondering if their is a solution to the problem yet.
Gostev
Chief Product Officer
Posts: 31455
Liked: 6646 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Incredibly slow Direc SAN restore

Post by Gostev »

I overheard recently that support now has a patch that switches thick disk restore mode to eager zeroed.
SyNtAxx
Expert
Posts: 149
Liked: 15 times
Joined: Jan 02, 2015 7:12 pm
Contact:

Re: Incredibly slow Direc SAN restore

Post by SyNtAxx »

Gostev wrote:I overheard recently that support now has a patch that switches thick disk restore mode to eager zeroed.
Fantastic! is it general release for public consumption, or do you have to ask nicely for it :) ?

Thanks.
Gostev
Chief Product Officer
Posts: 31455
Liked: 6646 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Incredibly slow Direc SAN restore

Post by Gostev »

With rare exceptions, you don't have to be nice to get hot fixes which are already in support's hands ;)
Taran
Veeam Software
Posts: 87
Liked: 10 times
Joined: Jul 16, 2013 5:05 am
Full Name: Taran
Contact:

Re: Incredibly slow Direc SAN restore

Post by Taran » 1 person likes this post

Here's an update for everyone. The hotfix supplied by support improved the restore speeds to 200MB/s. There is some further work happening, however this is a significant improvement.

Asking support nicely is the way to go to get the hotfix (being nice never hurts). It might be a good idea to reference this forum post.
MrDave
Novice
Posts: 3
Liked: never
Joined: Sep 02, 2015 6:50 am
Contact:

Re: Incredibly slow Direc SAN restore

Post by MrDave »

Worked a treat. We do believe however that we should be able to achieve speeds faster than 200MB/s as we have some SSD trays in our SAN. There is further investigations happening there.
Lasr
Influencer
Posts: 16
Liked: 4 times
Joined: Dec 23, 2014 2:22 pm
Contact:

Re: Incredibly slow Direc SAN restore

Post by Lasr »

Hi,

is this hotifx included in B&R 8.0 Update 3?


Regards
Lasr
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Incredibly slow Direc SAN restore

Post by foggy »

No, chances are this will be addressed in v9.
Gostev
Chief Product Officer
Posts: 31455
Liked: 6646 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Incredibly slow Direc SAN restore

Post by Gostev »

Based on hot fix testing, for v9 we are simply planning to make this a default behavior.
Right now, there is simply not enough time to build and test delivering this as an option.
SyNtAxx
Expert
Posts: 149
Liked: 15 times
Joined: Jan 02, 2015 7:12 pm
Contact:

Re: Incredibly slow Direc SAN restore

Post by SyNtAxx »

What is the path/hotfix number so I may request it?

-Nick
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Incredibly slow Direc SAN restore

Post by veremin »

Within an existing support ticket just refer to the Taran's post. I think it should be enough to get the said fix. Thanks.
Anders
Enthusiast
Posts: 58
Liked: 12 times
Joined: Sep 09, 2010 9:45 am
Full Name: Anders Lorensen
Contact:

Re: Incredibly slow Direc SAN restore

Post by Anders » 1 person likes this post

With a EMC VNX5300 attached with 8G FC I got these numbers
The VM I've testet this with is 500 GB in size. (1x 100 GB, 2x 200 GB VMDKs)
Direct SAN backups run at around 150 MB/s
Network backup runs at 70 MB/s (1G Network)
Restore via Network runs at about 70 MB/s (1G Network)
Restore via DirectSAN without patch runs at 25-45 MB/s
Restore via DirectSAN with the patch runs at about 99 MB/s
The problem with the patch is that it takes ages before the restore starts because it creates an empty zeroed VMDK file first, and that took longer than the entire VM Restore does via Network. We are pretty sure this is because we have VAAI disabled due to a bug in the EMC VNX firmware.

In the end we uninstalled the patch.

So making the patch the default in v9, without a way to disable this, will make people with VAAI disabled cry!

Also, remember to include your Veeam build number when requesting the patch via support - will speed up getting the patch by a day or two!
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Incredibly slow Direc SAN restore

Post by foggy »

There will be a registry value to switch the behavior. Btw, thanks for this feedback, much appreciated.
m1kkel
Enthusiast
Posts: 47
Liked: 1 time
Joined: Nov 06, 2014 8:01 pm
Full Name: Mikkel Nielsen
Contact:

[MERGED] Slow restore san mode - testing, weird result

Post by m1kkel »

Hello.

Just finishing setting up restore and backup via san mode. Had some issues with restore, because of lun's were write protected on my proxy server, which is windows server 2012R2, because of default san policy.

Anyway, the restore of an entire VM with only 1 restore point, was slow, around 77 MB / sec. Tried Hotadd, which was faster, around 90 MB / sec.

Here is the really weird part. Server is 2012R2 running storage spaces, 12 disk presented as jbod without raid controller, SATA 7,2K Mirror with deduplication. CPU is xeon something with 4 cores, and 32 GB memory. Now, i am copying a file internally on the server, from the V: drive (for veeam :-) ) to the C drive, which is a SSD disk (in mirror on another controller)
Copying a VBK file from the backupupjob folder called "DirectSan_test_Hostingtest" to the C: drive, i get a speed rating around 30-130 / sec. (size is around 30GB) Copying some VIB files from other folders around 17 GB in size, i get speed around 500-700 MB pr. second which is as expected. And some of the other VIB files from some of the other backup jobs are REALLY slow just copying internally from V: to C: and some are extremely fast.
Volume is NTFS, formatted in 64 KB cluster size, and formatted with the option for large drives.

This does not make any sense to me, is something fragmented, and maybe some of the chunks are placed randomly on the volume, and maybe some are aligned up nicely?

Settings in veeam for storage is local target, and optimal compression.
In veeam, i am not using the option for "Align backup file datablocks"

Suggestions?
Post Reply

Who is online

Users browsing this forum: Google [Bot], vladan.obradovic and 86 guests