-
- Enthusiast
- Posts: 78
- Liked: 4 times
- Joined: Jan 12, 2012 3:45 am
- Full Name: claudiofolu
- Contact:
Incredibly slow Direc SAN restore
Hello,
We are doing a restore with a Direct SAN proxy and the process rate never goes beyond 8MB/s
The logs shows that the proxy selection is correct, and that the access method is [san]. FibreChannel
The proxy has access to both LUNs, source and target. Different storages.
Any clues?
We are doing a restore with a Direct SAN proxy and the process rate never goes beyond 8MB/s
The logs shows that the proxy selection is correct, and that the access method is [san]. FibreChannel
The proxy has access to both LUNs, source and target. Different storages.
Any clues?
-
- Expert
- Posts: 149
- Liked: 15 times
- Joined: Jan 02, 2015 7:12 pm
- Contact:
Re: Incredible slow Direc SAN restore
I am trying to get direct san restore to even begin to work. I'd be interested in your settings on the luns as viewed from the proxy if you dont mind sharing.
-
- VP, Product Management
- Posts: 27377
- Liked: 2800 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: Incredible slow Direc SAN restore
@claudiofolu, if you try to restore via NBD or hotadd, what performance do you have? Also have you tried to restore to another LUN?
@SyNtAxx, LUNs should be presented in the same way as you do it when configure direct SAN mode for VM data retrieval.
@SyNtAxx, LUNs should be presented in the same way as you do it when configure direct SAN mode for VM data retrieval.
-
- Enthusiast
- Posts: 78
- Liked: 4 times
- Joined: Jan 12, 2012 3:45 am
- Full Name: claudiofolu
- Contact:
Re: Incredible slow Direc SAN restore
Hello Vitaly, I only have one LUN to test. it´s a 16TB LUN. (VMFS5)Vitaliy S. wrote:@claudiofolu, if you try to restore via NBD or hotadd, what performance do you have? Also have you tried to restore to another LUN?
with NBD: 17MB/s
-
- Expert
- Posts: 149
- Liked: 15 times
- Joined: Jan 02, 2015 7:12 pm
- Contact:
Re: Incredible slow Direc SAN restore
I should have been more clear, but I also didn't want to hijack the thread...Vitaliy S. wrote:@claudiofolu, if you try to restore via NBD or hotadd, what performance do you have? Also have you tried to restore to another LUN?
@SyNtAxx, LUNs should be presented in the same way as you do it when configure direct SAN mode for VM data retrieval.
My direct SAN backups work with out the need to present the luns to my proxy server. I suspect his is because I am using the 3par integration properties of Veeam. I have a single lun presented to the proxy server that I want to use for recoveries only. The proxy is a physical machine, with Windows 12012 R2 installed. The test lun in question is vmfs (esxi 5.5) as as viwed from the proxy 'offline' and 'read only'. It seems obvious that it needs to be 'writeable' and presumably 'online' but I've not found any explicit direction in Veeam manuals or online help.
Thanks,
Nick
-
- VP, Product Management
- Posts: 27377
- Liked: 2800 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: Incredible slow Direc SAN restore
I don't believe bringing disks on-line is the requirement. There are no instructions, as setting up the connection for restores is the same as configuring SAN LUNs for data retrieval. Be aware that SAN restores are only possible for thick disks. If it is not the case, then I would suggest looking into restore job log for further information.SyNtAxx wrote:It seems obvious that it needs to be 'writeable' and presumably 'online' but I've not found any explicit direction in Veeam manuals or online help.
BTW, you can confirm that your backup jobs can be run in Direct SAN Mode, by unchecking storage integration option. In order to do a quick test like that, just create any dummy VM (even without Guest OS) and try to back it up via direct SAN mode.SyNtAxx wrote:My direct SAN backups work with out the need to present the luns to my proxy server. I suspect his is because I am using the 3par integration properties of Veeam.
What about backup job performance via direct SAN mode? What are the rates and what it the bottleneck in this case? Additionally, do restores via hotadd also have the same performance as NBD mode?claudiofolu wrote:Hello Vitaly, I only have one LUN to test. it´s a 16TB LUN. (VMFS5)
with NBD: 17MB/s
-
- Enthusiast
- Posts: 78
- Liked: 4 times
- Joined: Jan 12, 2012 3:45 am
- Full Name: claudiofolu
- Contact:
Re: Incredible slow Direc SAN restore
Hello Vitaly,
We´ve opened a support case #00825830
Performance in backup jobs is excellent, 130MB/s
Could the size of the LUN be a problem?
We also checked is the lun was ReadOnly, and no
We´ve opened a support case #00825830
Performance in backup jobs is excellent, 130MB/s
Could the size of the LUN be a problem?
We also checked is the lun was ReadOnly, and no
-
- VP, Product Management
- Posts: 27377
- Liked: 2800 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: Incredible slow Direc SAN restore
Yes, the performance is indeed almost perfect. Is it a full run? The size of the LUN should not matter, let's see what our support team says after reviewing log files.
-
- Enthusiast
- Posts: 78
- Liked: 4 times
- Joined: Jan 12, 2012 3:45 am
- Full Name: claudiofolu
- Contact:
Re: Incredible slow Direc SAN restore
Support find that the target, in this case the Datastore we are using to restore, is at 99%
This is strage, since the Storage we are using to restore (EMC VNX 5300) is not running anything else, and the host is physical with 32GB of ram
How can we find the real problem here?
This is strage, since the Storage we are using to restore (EMC VNX 5300) is not running anything else, and the host is physical with 32GB of ram
How can we find the real problem here?
-
- VP, Product Management
- Posts: 27377
- Liked: 2800 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: Incredible slow Direc SAN restore
What rates do you have for hotadd restores? Is it comparable to network and direct SAN modes?
-
- Enthusiast
- Posts: 78
- Liked: 4 times
- Joined: Jan 12, 2012 3:45 am
- Full Name: claudiofolu
- Contact:
Re: Incredible slow Direc SAN restore
Didn`t test it. We don`t have a virtual proxy on the DRP site.
-
- Expert
- Posts: 149
- Liked: 15 times
- Joined: Jan 02, 2015 7:12 pm
- Contact:
Re: Incredible slow Direc SAN restore
I have gotten my direct san recoveries working. My question now is what rate to I use as the benchmark speed? On the Stats tab of the restore, im seeing 334MB/sec. On the Logs tab, the speed is indicated as 82MB/sec on the last vmdk being recovered. Its a bit confusing.
-Nick
-Nick
-
- VP, Product Management
- Posts: 27377
- Liked: 2800 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: Incredible slow Direc SAN restore
Nick, can you please share what you've done to increase the performance of the restore job? Did you restore multiple vmdks or the VM had more than 1 disk?
-
- Expert
- Posts: 149
- Liked: 15 times
- Joined: Jan 02, 2015 7:12 pm
- Contact:
Re: Incredible slow Direc SAN restore
The vm I am restoring had multiple vmdk files. One is 60gb and the other is 1.5tb. The SAN I am recovering to have 1400 drives on the disk tier I am using. Ive done nothing but present a test lun and make it writable from the proxies pint of view.
I will say, my test restore has not made any progress in hours. I am wondering the job is hung.
-Nick
I will say, my test restore has not made any progress in hours. I am wondering the job is hung.
-Nick
-
- Expert
- Posts: 149
- Liked: 15 times
- Joined: Jan 02, 2015 7:12 pm
- Contact:
Re: Incredible slow Direc SAN restore
I cancelled my test restore and restarted it. It hangs at the same place. 32% and the metters no longer count down. Time is still, data restored is still, etc. I think I'll have to open a case.
-Nick
-Nick
-
- VP, Product Management
- Posts: 27377
- Liked: 2800 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: Incredible slow Direc SAN restore
Yes, please and let us know your case ID. Thanks!
-
- Expert
- Posts: 149
- Liked: 15 times
- Joined: Jan 02, 2015 7:12 pm
- Contact:
Re: Incredible slow Direc SAN restore
Case # 00828648
-
- Veteran
- Posts: 361
- Liked: 109 times
- Joined: Dec 28, 2012 5:20 pm
- Full Name: Guido Meijers
- Contact:
Re: Incredible slow Direc SAN restore
I can confirm the san volume needs to put online by hand, and write access needs to be setup on san side. Also for direct SAN Backups disks need to be put online by hand (but not writable of course)
However we also cannot find a Reason for slow restore speeds. SAN is able to do 2GB/s+ and Backup repository is also able to at least read at a few hundred mb/s (72 7K4 SATA Disks in multiple Raid 5s striped in Windows). However we never get over 22MB/s, NDB restores are about the same. We could never find a reason Network is 10Gbit, SAN is FC connected.
However we also cannot find a Reason for slow restore speeds. SAN is able to do 2GB/s+ and Backup repository is also able to at least read at a few hundred mb/s (72 7K4 SATA Disks in multiple Raid 5s striped in Windows). However we never get over 22MB/s, NDB restores are about the same. We could never find a reason Network is 10Gbit, SAN is FC connected.
-
- VP, Product Management
- Posts: 27377
- Liked: 2800 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: Incredible slow Direc SAN restore
Guido , volumes should not be online, at least in my lab I have backup job running in a direct SAN mode with offline volumes. Let our support team look through your restore job logs, they might be able to find the reason for this slow performance.
-
- Service Provider
- Posts: 60
- Liked: 19 times
- Joined: Dec 23, 2014 4:04 pm
- Contact:
Re: Incredible slow Direc SAN restore
We are also using 3PAR integration. Is direct SAN restore possible in this configuration?
I actually doubt it since LUN's are only presented at the moment of backup and even then only snapshots not the original LUN...
Is Veeam smart enough to present the original LUN for a restore with 3PAR integration?
In my testing I only get NBD restores...
I actually doubt it since LUN's are only presented at the moment of backup and even then only snapshots not the original LUN...
Is Veeam smart enough to present the original LUN for a restore with 3PAR integration?
In my testing I only get NBD restores...
-
- Veeam Software
- Posts: 21139
- Liked: 2141 times
- Joined: Jul 11, 2011 10:22 am
- Full Name: Alexander Fogelson
- Contact:
Re: Incredible slow Direc SAN restore
If you have the LUN itself presented to the proxy server in the read-write mode, direct SAN restore will be available.
-
- Service Provider
- Posts: 60
- Liked: 19 times
- Joined: Dec 23, 2014 4:04 pm
- Contact:
Re: Incredible slow Direc SAN restore
Ok, will test with presenting the lun...
Other question, the documentation you linked to mentions the following:
The Direct SAN Access transport mode can be used to restore VMs with thick disks only
Do they need to be eager zeroed or is lazy sufficient?
Other question, the documentation you linked to mentions the following:
The Direct SAN Access transport mode can be used to restore VMs with thick disks only
Do they need to be eager zeroed or is lazy sufficient?
-
- Service Provider
- Posts: 60
- Liked: 19 times
- Joined: Dec 23, 2014 4:04 pm
- Contact:
Re: Incredible slow Direc SAN restore
Ok, just tested with presenting the LUN in read/write and can confirm this makes SAN restore available.
However the disk needs to be online for it to work. If the disk is offline it will attempt SAN restore but will failover to network:
Unable to execute write operation using advanced transport, failing over to network.
Not sure if this is a bug or expected behaviour but keeping my production lun's online makes me a little nervous...
Would be nice if we could keep them offline. Even better would be if Veeam could use the 3PAR storage integration to selectively present the production LUN when needed for a restore...
Direct SAN restore performance was a little disappointing.
80MB/s is not bad but it's the same as restoring over network.
Was hoping for an increase but I guess my bottleneck is somewhere else...
However the disk needs to be online for it to work. If the disk is offline it will attempt SAN restore but will failover to network:
Unable to execute write operation using advanced transport, failing over to network.
Not sure if this is a bug or expected behaviour but keeping my production lun's online makes me a little nervous...
Would be nice if we could keep them offline. Even better would be if Veeam could use the 3PAR storage integration to selectively present the production LUN when needed for a restore...
Direct SAN restore performance was a little disappointing.
80MB/s is not bad but it's the same as restoring over network.
Was hoping for an increase but I guess my bottleneck is somewhere else...
-
- VP, Product Management
- Posts: 27377
- Liked: 2800 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: Incredible slow Direc SAN restore
There is no need to be nervous since Veeam B&R server automatically sets SAN policy to offline mode and automount to disabled for all new volumes on the proxy server.WimVD wrote:Not sure if this is a bug or expected behaviour but keeping my production lun's online makes me a little nervous...
-
- Chief Product Officer
- Posts: 31806
- Liked: 7300 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: Incredible slow Direc SAN restore
This should not be the case?WimVD wrote:However the disk needs to be online for it to work.
-
- Veteran
- Posts: 361
- Liked: 109 times
- Joined: Dec 28, 2012 5:20 pm
- Full Name: Guido Meijers
- Contact:
Re: Incredible slow Direc SAN restore
I can also confirm disks need to be put online or job will fallback to ndb
@Vitaly we already had veeam support looking at this, they confirmed we need to put disks online first (not initialize of course!), but reason for slow restores could not be found.
We also changed a reg entry for local data movers but it didn't help much. We could have done further investigations but as a "one man show" i cannot spends week doing trail & error so we learned to live with it for now.
@Vitaly we already had veeam support looking at this, they confirmed we need to put disks online first (not initialize of course!), but reason for slow restores could not be found.
We also changed a reg entry for local data movers but it didn't help much. We could have done further investigations but as a "one man show" i cannot spends week doing trail & error so we learned to live with it for now.
-
- Veeam Software
- Posts: 21139
- Liked: 2141 times
- Joined: Jul 11, 2011 10:22 am
- Full Name: Alexander Fogelson
- Contact:
Re: Incredible slow Direc SAN restore
Both thick disks types are supported.WimVD wrote:Do they need to be eager zeroed or is lazy sufficient?
-
- Enthusiast
- Posts: 78
- Liked: 4 times
- Joined: Jan 12, 2012 3:45 am
- Full Name: claudiofolu
- Contact:
Re: Incredible slow Direc SAN restore
Hello everybody
I want to give you an update on this.
The problema was that some read cache on the VNX box was disabled due to a battery failure. After re-enable it, restore rate was around 100MB/s.
Regards,
I want to give you an update on this.
The problema was that some read cache on the VNX box was disabled due to a battery failure. After re-enable it, restore rate was around 100MB/s.
Regards,
-
- Expert
- Posts: 149
- Liked: 15 times
- Joined: Jan 02, 2015 7:12 pm
- Contact:
Re: Incredible slow Direc SAN restore
Ive been doing additional testing. My actual SAN restore rate is about 90MB/SEC. I have a problem though, large vm restores fail. I have many vms over 1TB and if I cant recover in the fastest way possible it is problematic. Anybody else have failed direct SAN restores on large vms?
-SyNtAxx
-SyNtAxx
-
- Service Provider
- Posts: 60
- Liked: 19 times
- Joined: Dec 23, 2014 4:04 pm
- Contact:
Re: Incredible slow Direc SAN restore
We have had issues with large disk restores (VM File) over network failing with "NFC server is busy"
This was the whole reason we are looking at direct SAN restores. Haven't tested large SAN restores yet so hopefully they are more stable.
This was the whole reason we are looking at direct SAN restores. Haven't tested large SAN restores yet so hopefully they are more stable.
Who is online
Users browsing this forum: No registered users and 57 guests