Discussions specific to the VMware vSphere hypervisor
claudiofolu
Enthusiast
Posts: 52
Liked: 2 times
Joined: Jan 12, 2012 3:45 am
Full Name: claudiofolu
Contact:

Incredibly slow Direc SAN restore

Post by claudiofolu » Mar 06, 2015 7:04 pm

Hello,
We are doing a restore with a Direct SAN proxy and the process rate never goes beyond 8MB/s

The logs shows that the proxy selection is correct, and that the access method is [san]. FibreChannel
The proxy has access to both LUNs, source and target. Different storages.
Any clues?

SyNtAxx
Expert
Posts: 148
Liked: 15 times
Joined: Jan 02, 2015 7:12 pm
Contact:

Re: Incredible slow Direc SAN restore

Post by SyNtAxx » Mar 06, 2015 7:07 pm

I am trying to get direct san restore to even begin to work. I'd be interested in your settings on the luns as viewed from the proxy if you dont mind sharing.

Vitaliy S.
Product Manager
Posts: 23070
Liked: 1582 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Incredible slow Direc SAN restore

Post by Vitaliy S. » Mar 06, 2015 7:24 pm

@claudiofolu, if you try to restore via NBD or hotadd, what performance do you have? Also have you tried to restore to another LUN?

@SyNtAxx, LUNs should be presented in the same way as you do it when configure direct SAN mode for VM data retrieval.

claudiofolu
Enthusiast
Posts: 52
Liked: 2 times
Joined: Jan 12, 2012 3:45 am
Full Name: claudiofolu
Contact:

Re: Incredible slow Direc SAN restore

Post by claudiofolu » Mar 06, 2015 7:38 pm

Vitaliy S. wrote:@claudiofolu, if you try to restore via NBD or hotadd, what performance do you have? Also have you tried to restore to another LUN?
Hello Vitaly, I only have one LUN to test. it´s a 16TB LUN. (VMFS5)
with NBD: 17MB/s

SyNtAxx
Expert
Posts: 148
Liked: 15 times
Joined: Jan 02, 2015 7:12 pm
Contact:

Re: Incredible slow Direc SAN restore

Post by SyNtAxx » Mar 06, 2015 7:48 pm

Vitaliy S. wrote:@claudiofolu, if you try to restore via NBD or hotadd, what performance do you have? Also have you tried to restore to another LUN?

@SyNtAxx, LUNs should be presented in the same way as you do it when configure direct SAN mode for VM data retrieval.
I should have been more clear, but I also didn't want to hijack the thread...

My direct SAN backups work with out the need to present the luns to my proxy server. I suspect his is because I am using the 3par integration properties of Veeam. I have a single lun presented to the proxy server that I want to use for recoveries only. The proxy is a physical machine, with Windows 12012 R2 installed. The test lun in question is vmfs (esxi 5.5) as as viwed from the proxy 'offline' and 'read only'. It seems obvious that it needs to be 'writeable' and presumably 'online' but I've not found any explicit direction in Veeam manuals or online help.

Thanks,

Nick

Vitaliy S.
Product Manager
Posts: 23070
Liked: 1582 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Incredible slow Direc SAN restore

Post by Vitaliy S. » Mar 07, 2015 3:56 pm

SyNtAxx wrote:It seems obvious that it needs to be 'writeable' and presumably 'online' but I've not found any explicit direction in Veeam manuals or online help.
I don't believe bringing disks on-line is the requirement. There are no instructions, as setting up the connection for restores is the same as configuring SAN LUNs for data retrieval. Be aware that SAN restores are only possible for thick disks. If it is not the case, then I would suggest looking into restore job log for further information.
SyNtAxx wrote:My direct SAN backups work with out the need to present the luns to my proxy server. I suspect his is because I am using the 3par integration properties of Veeam.
BTW, you can confirm that your backup jobs can be run in Direct SAN Mode, by unchecking storage integration option. In order to do a quick test like that, just create any dummy VM (even without Guest OS) and try to back it up via direct SAN mode.
claudiofolu wrote:Hello Vitaly, I only have one LUN to test. it´s a 16TB LUN. (VMFS5)
with NBD: 17MB/s
What about backup job performance via direct SAN mode? What are the rates and what it the bottleneck in this case? Additionally, do restores via hotadd also have the same performance as NBD mode?

claudiofolu
Enthusiast
Posts: 52
Liked: 2 times
Joined: Jan 12, 2012 3:45 am
Full Name: claudiofolu
Contact:

Re: Incredible slow Direc SAN restore

Post by claudiofolu » Mar 09, 2015 6:38 pm

Hello Vitaly,
We´ve opened a support case #00825830
Performance in backup jobs is excellent, 130MB/s
Could the size of the LUN be a problem?
We also checked is the lun was ReadOnly, and no

Vitaliy S.
Product Manager
Posts: 23070
Liked: 1582 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Incredible slow Direc SAN restore

Post by Vitaliy S. » Mar 09, 2015 6:59 pm

Yes, the performance is indeed almost perfect. Is it a full run? The size of the LUN should not matter, let's see what our support team says after reviewing log files.

claudiofolu
Enthusiast
Posts: 52
Liked: 2 times
Joined: Jan 12, 2012 3:45 am
Full Name: claudiofolu
Contact:

Re: Incredible slow Direc SAN restore

Post by claudiofolu » Mar 10, 2015 12:59 pm

Support find that the target, in this case the Datastore we are using to restore, is at 99%
This is strage, since the Storage we are using to restore (EMC VNX 5300) is not running anything else, and the host is physical with 32GB of ram

How can we find the real problem here?

Vitaliy S.
Product Manager
Posts: 23070
Liked: 1582 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Incredible slow Direc SAN restore

Post by Vitaliy S. » Mar 10, 2015 2:04 pm

What rates do you have for hotadd restores? Is it comparable to network and direct SAN modes?

claudiofolu
Enthusiast
Posts: 52
Liked: 2 times
Joined: Jan 12, 2012 3:45 am
Full Name: claudiofolu
Contact:

Re: Incredible slow Direc SAN restore

Post by claudiofolu » Mar 10, 2015 3:07 pm

Didn`t test it. We don`t have a virtual proxy on the DRP site.

SyNtAxx
Expert
Posts: 148
Liked: 15 times
Joined: Jan 02, 2015 7:12 pm
Contact:

Re: Incredible slow Direc SAN restore

Post by SyNtAxx » Mar 10, 2015 4:15 pm

I have gotten my direct san recoveries working. My question now is what rate to I use as the benchmark speed? On the Stats tab of the restore, im seeing 334MB/sec. On the Logs tab, the speed is indicated as 82MB/sec on the last vmdk being recovered. Its a bit confusing.

-Nick

Vitaliy S.
Product Manager
Posts: 23070
Liked: 1582 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Incredible slow Direc SAN restore

Post by Vitaliy S. » Mar 10, 2015 5:06 pm

Nick, can you please share what you've done to increase the performance of the restore job? Did you restore multiple vmdks or the VM had more than 1 disk?

SyNtAxx
Expert
Posts: 148
Liked: 15 times
Joined: Jan 02, 2015 7:12 pm
Contact:

Re: Incredible slow Direc SAN restore

Post by SyNtAxx » Mar 10, 2015 5:54 pm

The vm I am restoring had multiple vmdk files. One is 60gb and the other is 1.5tb. The SAN I am recovering to have 1400 drives on the disk tier I am using. Ive done nothing but present a test lun and make it writable from the proxies pint of view.

I will say, my test restore has not made any progress in hours. I am wondering the job is hung.

-Nick

SyNtAxx
Expert
Posts: 148
Liked: 15 times
Joined: Jan 02, 2015 7:12 pm
Contact:

Re: Incredible slow Direc SAN restore

Post by SyNtAxx » Mar 10, 2015 10:13 pm

I cancelled my test restore and restarted it. It hangs at the same place. 32% and the metters no longer count down. Time is still, data restored is still, etc. I think I'll have to open a case.

-Nick

Vitaliy S.
Product Manager
Posts: 23070
Liked: 1582 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Incredible slow Direc SAN restore

Post by Vitaliy S. » Mar 11, 2015 12:09 pm

Yes, please and let us know your case ID. Thanks!

SyNtAxx
Expert
Posts: 148
Liked: 15 times
Joined: Jan 02, 2015 7:12 pm
Contact:

Re: Incredible slow Direc SAN restore

Post by SyNtAxx » Mar 11, 2015 1:35 pm

Case # 00828648

Delo123
Expert
Posts: 361
Liked: 109 times
Joined: Dec 28, 2012 5:20 pm
Full Name: Guido Meijers
Contact:

Re: Incredible slow Direc SAN restore

Post by Delo123 » Mar 11, 2015 5:42 pm

I can confirm the san volume needs to put online by hand, and write access needs to be setup on san side. Also for direct SAN Backups disks need to be put online by hand (but not writable of course)
However we also cannot find a Reason for slow restore speeds. SAN is able to do 2GB/s+ and Backup repository is also able to at least read at a few hundred mb/s (72 7K4 SATA Disks in multiple Raid 5s striped in Windows). However we never get over 22MB/s, NDB restores are about the same. We could never find a reason :( Network is 10Gbit, SAN is FC connected.

Vitaliy S.
Product Manager
Posts: 23070
Liked: 1582 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Incredible slow Direc SAN restore

Post by Vitaliy S. » Mar 12, 2015 10:26 am

Guido , volumes should not be online, at least in my lab I have backup job running in a direct SAN mode with offline volumes. Let our support team look through your restore job logs, they might be able to find the reason for this slow performance.

WimVD
Service Provider
Posts: 53
Liked: 18 times
Joined: Dec 23, 2014 4:04 pm
Contact:

Re: Incredible slow Direc SAN restore

Post by WimVD » Mar 13, 2015 1:55 pm

We are also using 3PAR integration. Is direct SAN restore possible in this configuration?
I actually doubt it since LUN's are only presented at the moment of backup and even then only snapshots not the original LUN...
Is Veeam smart enough to present the original LUN for a restore with 3PAR integration?

In my testing I only get NBD restores...

foggy
Veeam Software
Posts: 18366
Liked: 1576 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Incredible slow Direc SAN restore

Post by foggy » Mar 13, 2015 2:11 pm

If you have the LUN itself presented to the proxy server in the read-write mode, direct SAN restore will be available.

WimVD
Service Provider
Posts: 53
Liked: 18 times
Joined: Dec 23, 2014 4:04 pm
Contact:

Re: Incredible slow Direc SAN restore

Post by WimVD » Mar 13, 2015 3:38 pm

Ok, will test with presenting the lun...
Other question, the documentation you linked to mentions the following:
The Direct SAN Access transport mode can be used to restore VMs with thick disks only

Do they need to be eager zeroed or is lazy sufficient?

WimVD
Service Provider
Posts: 53
Liked: 18 times
Joined: Dec 23, 2014 4:04 pm
Contact:

Re: Incredible slow Direc SAN restore

Post by WimVD » Mar 13, 2015 4:40 pm

Ok, just tested with presenting the LUN in read/write and can confirm this makes SAN restore available.
However the disk needs to be online for it to work. If the disk is offline it will attempt SAN restore but will failover to network:
Unable to execute write operation using advanced transport, failing over to network.

Not sure if this is a bug or expected behaviour but keeping my production lun's online makes me a little nervous...
Would be nice if we could keep them offline. Even better would be if Veeam could use the 3PAR storage integration to selectively present the production LUN when needed for a restore...

Direct SAN restore performance was a little disappointing.
80MB/s is not bad but it's the same as restoring over network.
Was hoping for an increase but I guess my bottleneck is somewhere else...

Vitaliy S.
Product Manager
Posts: 23070
Liked: 1582 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Incredible slow Direc SAN restore

Post by Vitaliy S. » Mar 13, 2015 8:43 pm

WimVD wrote:Not sure if this is a bug or expected behaviour but keeping my production lun's online makes me a little nervous...
There is no need to be nervous since Veeam B&R server automatically sets SAN policy to offline mode and automount to disabled for all new volumes on the proxy server.

Gostev
SVP, Product Management
Posts: 24940
Liked: 3622 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Incredible slow Direc SAN restore

Post by Gostev » Mar 13, 2015 9:45 pm

WimVD wrote:However the disk needs to be online for it to work.
This should not be the case?

Delo123
Expert
Posts: 361
Liked: 109 times
Joined: Dec 28, 2012 5:20 pm
Full Name: Guido Meijers
Contact:

Re: Incredible slow Direc SAN restore

Post by Delo123 » Mar 16, 2015 8:37 am

I can also confirm disks need to be put online or job will fallback to ndb
@Vitaly we already had veeam support looking at this, they confirmed we need to put disks online first (not initialize of course!), but reason for slow restores could not be found.
We also changed a reg entry for local data movers but it didn't help much. We could have done further investigations but as a "one man show" i cannot spends week doing trail & error so we learned to live with it for now.

foggy
Veeam Software
Posts: 18366
Liked: 1576 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Incredible slow Direc SAN restore

Post by foggy » Mar 16, 2015 10:02 am

WimVD wrote:Do they need to be eager zeroed or is lazy sufficient?
Both thick disks types are supported.

claudiofolu
Enthusiast
Posts: 52
Liked: 2 times
Joined: Jan 12, 2012 3:45 am
Full Name: claudiofolu
Contact:

Re: Incredible slow Direc SAN restore

Post by claudiofolu » Mar 16, 2015 1:04 pm 1 person likes this post

Hello everybody
I want to give you an update on this.
The problema was that some read cache on the VNX box was disabled due to a battery failure. After re-enable it, restore rate was around 100MB/s.

Regards,

SyNtAxx
Expert
Posts: 148
Liked: 15 times
Joined: Jan 02, 2015 7:12 pm
Contact:

Re: Incredible slow Direc SAN restore

Post by SyNtAxx » Mar 26, 2015 2:33 pm

Ive been doing additional testing. My actual SAN restore rate is about 90MB/SEC. I have a problem though, large vm restores fail. I have many vms over 1TB and if I cant recover in the fastest way possible it is problematic. Anybody else have failed direct SAN restores on large vms?

-SyNtAxx

WimVD
Service Provider
Posts: 53
Liked: 18 times
Joined: Dec 23, 2014 4:04 pm
Contact:

Re: Incredible slow Direc SAN restore

Post by WimVD » Mar 27, 2015 10:12 am

We have had issues with large disk restores (VM File) over network failing with "NFC server is busy"
This was the whole reason we are looking at direct SAN restores. Haven't tested large SAN restores yet so hopefully they are more stable.

Post Reply

Who is online

Users browsing this forum: No registered users and 12 guests