Incredibly slow Direc SAN restore

VMware specific discussions

Incredibly slow Direc SAN restore

Veeam Logoby claudiofolu » Fri Mar 06, 2015 7:04 pm

Hello,
We are doing a restore with a Direct SAN proxy and the process rate never goes beyond 8MB/s

The logs shows that the proxy selection is correct, and that the access method is [san]. FibreChannel
The proxy has access to both LUNs, source and target. Different storages.
Any clues?
claudiofolu
Enthusiast
 
Posts: 42
Liked: 1 time
Joined: Thu Jan 12, 2012 3:45 am
Full Name: claudiofolu

Re: Incredible slow Direc SAN restore

Veeam Logoby SyNtAxx » Fri Mar 06, 2015 7:07 pm

I am trying to get direct san restore to even begin to work. I'd be interested in your settings on the luns as viewed from the proxy if you dont mind sharing.
SyNtAxx
Expert
 
Posts: 127
Liked: 14 times
Joined: Fri Jan 02, 2015 7:12 pm

Re: Incredible slow Direc SAN restore

Veeam Logoby Vitaliy S. » Fri Mar 06, 2015 7:24 pm

@claudiofolu, if you try to restore via NBD or hotadd, what performance do you have? Also have you tried to restore to another LUN?

@SyNtAxx, LUNs should be presented in the same way as you do it when configure direct SAN mode for VM data retrieval.
Vitaliy S.
Veeam Software
 
Posts: 19558
Liked: 1102 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: Incredible slow Direc SAN restore

Veeam Logoby claudiofolu » Fri Mar 06, 2015 7:38 pm

Vitaliy S. wrote:@claudiofolu, if you try to restore via NBD or hotadd, what performance do you have? Also have you tried to restore to another LUN?


Hello Vitaly, I only have one LUN to test. it´s a 16TB LUN. (VMFS5)
with NBD: 17MB/s
claudiofolu
Enthusiast
 
Posts: 42
Liked: 1 time
Joined: Thu Jan 12, 2012 3:45 am
Full Name: claudiofolu

Re: Incredible slow Direc SAN restore

Veeam Logoby SyNtAxx » Fri Mar 06, 2015 7:48 pm

Vitaliy S. wrote:@claudiofolu, if you try to restore via NBD or hotadd, what performance do you have? Also have you tried to restore to another LUN?

@SyNtAxx, LUNs should be presented in the same way as you do it when configure direct SAN mode for VM data retrieval.


I should have been more clear, but I also didn't want to hijack the thread...

My direct SAN backups work with out the need to present the luns to my proxy server. I suspect his is because I am using the 3par integration properties of Veeam. I have a single lun presented to the proxy server that I want to use for recoveries only. The proxy is a physical machine, with Windows 12012 R2 installed. The test lun in question is vmfs (esxi 5.5) as as viwed from the proxy 'offline' and 'read only'. It seems obvious that it needs to be 'writeable' and presumably 'online' but I've not found any explicit direction in Veeam manuals or online help.

Thanks,

Nick
SyNtAxx
Expert
 
Posts: 127
Liked: 14 times
Joined: Fri Jan 02, 2015 7:12 pm

Re: Incredible slow Direc SAN restore

Veeam Logoby Vitaliy S. » Sat Mar 07, 2015 3:56 pm

SyNtAxx wrote:It seems obvious that it needs to be 'writeable' and presumably 'online' but I've not found any explicit direction in Veeam manuals or online help.

I don't believe bringing disks on-line is the requirement. There are no instructions, as setting up the connection for restores is the same as configuring SAN LUNs for data retrieval. Be aware that SAN restores are only possible for thick disks. If it is not the case, then I would suggest looking into restore job log for further information.

SyNtAxx wrote:My direct SAN backups work with out the need to present the luns to my proxy server. I suspect his is because I am using the 3par integration properties of Veeam.

BTW, you can confirm that your backup jobs can be run in Direct SAN Mode, by unchecking storage integration option. In order to do a quick test like that, just create any dummy VM (even without Guest OS) and try to back it up via direct SAN mode.

claudiofolu wrote:Hello Vitaly, I only have one LUN to test. it´s a 16TB LUN. (VMFS5)
with NBD: 17MB/s

What about backup job performance via direct SAN mode? What are the rates and what it the bottleneck in this case? Additionally, do restores via hotadd also have the same performance as NBD mode?
Vitaliy S.
Veeam Software
 
Posts: 19558
Liked: 1102 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: Incredible slow Direc SAN restore

Veeam Logoby claudiofolu » Mon Mar 09, 2015 6:38 pm

Hello Vitaly,
We´ve opened a support case #00825830
Performance in backup jobs is excellent, 130MB/s
Could the size of the LUN be a problem?
We also checked is the lun was ReadOnly, and no
claudiofolu
Enthusiast
 
Posts: 42
Liked: 1 time
Joined: Thu Jan 12, 2012 3:45 am
Full Name: claudiofolu

Re: Incredible slow Direc SAN restore

Veeam Logoby Vitaliy S. » Mon Mar 09, 2015 6:59 pm

Yes, the performance is indeed almost perfect. Is it a full run? The size of the LUN should not matter, let's see what our support team says after reviewing log files.
Vitaliy S.
Veeam Software
 
Posts: 19558
Liked: 1102 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: Incredible slow Direc SAN restore

Veeam Logoby claudiofolu » Tue Mar 10, 2015 12:59 pm

Support find that the target, in this case the Datastore we are using to restore, is at 99%
This is strage, since the Storage we are using to restore (EMC VNX 5300) is not running anything else, and the host is physical with 32GB of ram

How can we find the real problem here?
claudiofolu
Enthusiast
 
Posts: 42
Liked: 1 time
Joined: Thu Jan 12, 2012 3:45 am
Full Name: claudiofolu

Re: Incredible slow Direc SAN restore

Veeam Logoby Vitaliy S. » Tue Mar 10, 2015 2:04 pm

What rates do you have for hotadd restores? Is it comparable to network and direct SAN modes?
Vitaliy S.
Veeam Software
 
Posts: 19558
Liked: 1102 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: Incredible slow Direc SAN restore

Veeam Logoby claudiofolu » Tue Mar 10, 2015 3:07 pm

Didn`t test it. We don`t have a virtual proxy on the DRP site.
claudiofolu
Enthusiast
 
Posts: 42
Liked: 1 time
Joined: Thu Jan 12, 2012 3:45 am
Full Name: claudiofolu

Re: Incredible slow Direc SAN restore

Veeam Logoby SyNtAxx » Tue Mar 10, 2015 4:15 pm

I have gotten my direct san recoveries working. My question now is what rate to I use as the benchmark speed? On the Stats tab of the restore, im seeing 334MB/sec. On the Logs tab, the speed is indicated as 82MB/sec on the last vmdk being recovered. Its a bit confusing.

-Nick
SyNtAxx
Expert
 
Posts: 127
Liked: 14 times
Joined: Fri Jan 02, 2015 7:12 pm

Re: Incredible slow Direc SAN restore

Veeam Logoby Vitaliy S. » Tue Mar 10, 2015 5:06 pm

Nick, can you please share what you've done to increase the performance of the restore job? Did you restore multiple vmdks or the VM had more than 1 disk?
Vitaliy S.
Veeam Software
 
Posts: 19558
Liked: 1102 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: Incredible slow Direc SAN restore

Veeam Logoby SyNtAxx » Tue Mar 10, 2015 5:54 pm

The vm I am restoring had multiple vmdk files. One is 60gb and the other is 1.5tb. The SAN I am recovering to have 1400 drives on the disk tier I am using. Ive done nothing but present a test lun and make it writable from the proxies pint of view.

I will say, my test restore has not made any progress in hours. I am wondering the job is hung.

-Nick
SyNtAxx
Expert
 
Posts: 127
Liked: 14 times
Joined: Fri Jan 02, 2015 7:12 pm

Re: Incredible slow Direc SAN restore

Veeam Logoby SyNtAxx » Tue Mar 10, 2015 10:13 pm

I cancelled my test restore and restarted it. It hangs at the same place. 32% and the metters no longer count down. Time is still, data restored is still, etc. I think I'll have to open a case.

-Nick
SyNtAxx
Expert
 
Posts: 127
Liked: 14 times
Joined: Fri Jan 02, 2015 7:12 pm

Next

Return to VMware vSphere



Who is online

Users browsing this forum: vishalgupta, Yahoo [Bot] and 11 guests