Incredibly slow Direc SAN restore

VMware specific discussions

Re: Incredible slow Direc SAN restore

Veeam Logoby Vitaliy S. » Wed Mar 11, 2015 12:09 pm

Yes, please and let us know your case ID. Thanks!
Vitaliy S.
Veeam Software
 
Posts: 19564
Liked: 1104 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: Incredible slow Direc SAN restore

Veeam Logoby SyNtAxx » Wed Mar 11, 2015 1:35 pm

Case # 00828648
SyNtAxx
Expert
 
Posts: 127
Liked: 14 times
Joined: Fri Jan 02, 2015 7:12 pm

Re: Incredible slow Direc SAN restore

Veeam Logoby Delo123 » Wed Mar 11, 2015 5:42 pm

I can confirm the san volume needs to put online by hand, and write access needs to be setup on san side. Also for direct SAN Backups disks need to be put online by hand (but not writable of course)
However we also cannot find a Reason for slow restore speeds. SAN is able to do 2GB/s+ and Backup repository is also able to at least read at a few hundred mb/s (72 7K4 SATA Disks in multiple Raid 5s striped in Windows). However we never get over 22MB/s, NDB restores are about the same. We could never find a reason :( Network is 10Gbit, SAN is FC connected.
Delo123
Expert
 
Posts: 351
Liked: 97 times
Joined: Fri Dec 28, 2012 5:20 pm
Full Name: Guido Meijers

Re: Incredible slow Direc SAN restore

Veeam Logoby Vitaliy S. » Thu Mar 12, 2015 10:26 am

Guido , volumes should not be online, at least in my lab I have backup job running in a direct SAN mode with offline volumes. Let our support team look through your restore job logs, they might be able to find the reason for this slow performance.
Vitaliy S.
Veeam Software
 
Posts: 19564
Liked: 1104 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: Incredible slow Direc SAN restore

Veeam Logoby WimVD » Fri Mar 13, 2015 1:55 pm

We are also using 3PAR integration. Is direct SAN restore possible in this configuration?
I actually doubt it since LUN's are only presented at the moment of backup and even then only snapshots not the original LUN...
Is Veeam smart enough to present the original LUN for a restore with 3PAR integration?

In my testing I only get NBD restores...
WimVD
Service Provider
 
Posts: 48
Liked: 10 times
Joined: Tue Dec 23, 2014 4:04 pm

Re: Incredible slow Direc SAN restore

Veeam Logoby foggy » Fri Mar 13, 2015 2:11 pm

If you have the LUN itself presented to the proxy server in the read-write mode, direct SAN restore will be available.
foggy
Veeam Software
 
Posts: 14743
Liked: 1081 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Incredible slow Direc SAN restore

Veeam Logoby WimVD » Fri Mar 13, 2015 3:38 pm

Ok, will test with presenting the lun...
Other question, the documentation you linked to mentions the following:
The Direct SAN Access transport mode can be used to restore VMs with thick disks only

Do they need to be eager zeroed or is lazy sufficient?
WimVD
Service Provider
 
Posts: 48
Liked: 10 times
Joined: Tue Dec 23, 2014 4:04 pm

Re: Incredible slow Direc SAN restore

Veeam Logoby WimVD » Fri Mar 13, 2015 4:40 pm

Ok, just tested with presenting the LUN in read/write and can confirm this makes SAN restore available.
However the disk needs to be online for it to work. If the disk is offline it will attempt SAN restore but will failover to network:
Unable to execute write operation using advanced transport, failing over to network.

Not sure if this is a bug or expected behaviour but keeping my production lun's online makes me a little nervous...
Would be nice if we could keep them offline. Even better would be if Veeam could use the 3PAR storage integration to selectively present the production LUN when needed for a restore...

Direct SAN restore performance was a little disappointing.
80MB/s is not bad but it's the same as restoring over network.
Was hoping for an increase but I guess my bottleneck is somewhere else...
WimVD
Service Provider
 
Posts: 48
Liked: 10 times
Joined: Tue Dec 23, 2014 4:04 pm

Re: Incredible slow Direc SAN restore

Veeam Logoby Vitaliy S. » Fri Mar 13, 2015 8:43 pm

WimVD wrote:Not sure if this is a bug or expected behaviour but keeping my production lun's online makes me a little nervous...

There is no need to be nervous since Veeam B&R server automatically sets SAN policy to offline mode and automount to disabled for all new volumes on the proxy server.
Vitaliy S.
Veeam Software
 
Posts: 19564
Liked: 1104 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: Incredible slow Direc SAN restore

Veeam Logoby Gostev » Fri Mar 13, 2015 9:45 pm

WimVD wrote:However the disk needs to be online for it to work.

This should not be the case?
Gostev
Veeam Software
 
Posts: 21396
Liked: 2349 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: Incredible slow Direc SAN restore

Veeam Logoby Delo123 » Mon Mar 16, 2015 8:37 am

I can also confirm disks need to be put online or job will fallback to ndb
@Vitaly we already had veeam support looking at this, they confirmed we need to put disks online first (not initialize of course!), but reason for slow restores could not be found.
We also changed a reg entry for local data movers but it didn't help much. We could have done further investigations but as a "one man show" i cannot spends week doing trail & error so we learned to live with it for now.
Delo123
Expert
 
Posts: 351
Liked: 97 times
Joined: Fri Dec 28, 2012 5:20 pm
Full Name: Guido Meijers

Re: Incredible slow Direc SAN restore

Veeam Logoby foggy » Mon Mar 16, 2015 10:02 am

WimVD wrote:Do they need to be eager zeroed or is lazy sufficient?

Both thick disks types are supported.
foggy
Veeam Software
 
Posts: 14743
Liked: 1081 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Incredible slow Direc SAN restore

Veeam Logoby claudiofolu » Mon Mar 16, 2015 1:04 pm 1 person likes this post

Hello everybody
I want to give you an update on this.
The problema was that some read cache on the VNX box was disabled due to a battery failure. After re-enable it, restore rate was around 100MB/s.

Regards,
claudiofolu
Enthusiast
 
Posts: 42
Liked: 1 time
Joined: Thu Jan 12, 2012 3:45 am
Full Name: claudiofolu

Re: Incredible slow Direc SAN restore

Veeam Logoby SyNtAxx » Thu Mar 26, 2015 2:33 pm

Ive been doing additional testing. My actual SAN restore rate is about 90MB/SEC. I have a problem though, large vm restores fail. I have many vms over 1TB and if I cant recover in the fastest way possible it is problematic. Anybody else have failed direct SAN restores on large vms?

-SyNtAxx
SyNtAxx
Expert
 
Posts: 127
Liked: 14 times
Joined: Fri Jan 02, 2015 7:12 pm

Re: Incredible slow Direc SAN restore

Veeam Logoby WimVD » Fri Mar 27, 2015 10:12 am

We have had issues with large disk restores (VM File) over network failing with "NFC server is busy"
This was the whole reason we are looking at direct SAN restores. Haven't tested large SAN restores yet so hopefully they are more stable.
WimVD
Service Provider
 
Posts: 48
Liked: 10 times
Joined: Tue Dec 23, 2014 4:04 pm

PreviousNext

Return to VMware vSphere



Who is online

Users browsing this forum: No registered users and 7 guests