Incredibly slow Direc SAN restore

VMware specific discussions

Re: Incredible slow Direc SAN restore

Veeam Logoby Vitaliy S. » Fri Mar 27, 2015 11:14 am

What about using Instant VM recovery and then use migration jobs to bring VM back to production?
Vitaliy S.
Veeam Software
 
Posts: 19570
Liked: 1104 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: Incredible slow Direc SAN restore

Veeam Logoby SyNtAxx » Tue Mar 31, 2015 4:18 pm

How about the feature work correctly? My issue is currently at Tier 3 support.
SyNtAxx
Expert
 
Posts: 127
Liked: 14 times
Joined: Fri Jan 02, 2015 7:12 pm

Re: Incredible slow Direc SAN restore

Veeam Logoby Vitaliy S. » Tue Mar 31, 2015 5:03 pm

This doesn't mean I said stop troubleshooting it with our support team, I was just suggesting all available options ;)
Vitaliy S.
Veeam Software
 
Posts: 19570
Liked: 1104 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: Incredible slow Direc SAN restore

Veeam Logoby SyNtAxx » Tue Mar 31, 2015 5:13 pm

I understand, and apologize for the short reply. I am a bit frustrated. It seems like every time I go to test a new feature of the product I hit a wall. Small SAN restores complete, though large vms do not. Currently I've worked backwards from 1.5tb to a 900gb vm and it fails. I just got off the phone with support for yet another issue with Linux FLR not working. I get FIPs errors, and the appliance doesn't deploy. I've spent a lot of time convincing management this was the solution to go with and we've invested significantly in the product, 102 enterprise plus licensing and counting. I do my best to self resolve before I open a case so as not to waste an engineers time. So I've got a lot riding on this product and it is a critical application for us as I am sure it is for everybody else here.
SyNtAxx
Expert
 
Posts: 127
Liked: 14 times
Joined: Fri Jan 02, 2015 7:12 pm

Re: Incredible slow Direc SAN restore

Veeam Logoby chjones » Thu Apr 02, 2015 1:07 am

I can confirm as well that not bringing the SAN disks online before attempting a restore will fallover to network mode with the warning:

"Unable to execute write operation using advanced transport, failing over to network mode..."

We re-ran the restore after bringing the disks online in Windows Disk Management, and the restores start directly over the SAN. This just doesn't seem good design since the Veeam Proxy disables automount on disks, but can't use them in that state during a restore.
chjones
Enthusiast
 
Posts: 83
Liked: 25 times
Joined: Tue Oct 30, 2012 7:53 pm
Full Name: Chris Jones

Re: Incredible slow Direc SAN restore

Veeam Logoby SyNtAxx » Thu Apr 02, 2015 1:50 pm

Case #00828648 Open just short of a month so far. No resolve. Very large vms are a fail on restore every time. Has anybody successfully restored a 900gb+ vm successfully on version 8.0 Patch1 using any transport method? I've run into just about every known bug so far, requiring hotfixes to fix hotfixes. We are fortunate to have a very capable collection of hardware, but not sure why I am running into so many issues. I can admit to not being fully aware of all the nuances of the software features. However, at the core the configuration is good.

Environment:

Veeam Backup manager: vm @ 8 core 16gb ram and no proxy services, strictly a job manager.
proxy1: HP DL 580G7 4 procs @ 6 core + HT and 32gb ram. 4gb fiber hba to fabric. Teamed 1gE @ LACP. Win2k12R2
proxy2: HP BL460c g8: 2 procs @ 10 core + HT and 512gb ram and 2 @ 10gE for network and 2 @ 8gbps FC HBA. Win2k12R2 Blade chassis is 8x8gbps direct connected to SAN Frame and 40gE to the core network.

SAN:

3par 10800 fully loaded.
8 controllers
1700 drives.


Nick
SyNtAxx
Expert
 
Posts: 127
Liked: 14 times
Joined: Fri Jan 02, 2015 7:12 pm

Re: Incredible slow Direc SAN restore

Veeam Logoby Gostev » Thu Apr 02, 2015 7:42 pm

SyNtAxx wrote:I've run into just about every known bug so far

Looking at your case history, this indeed looks to be the case...
I've escalated this particular issue for you though.
Gostev
Veeam Software
 
Posts: 21396
Liked: 2350 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: Incredible slow Direc SAN restore

Veeam Logoby SyNtAxx » Thu Apr 02, 2015 9:53 pm

Gostev,

thank you.
SyNtAxx
Expert
 
Posts: 127
Liked: 14 times
Joined: Fri Jan 02, 2015 7:12 pm

Re: Incredible slow Direc SAN restore

Veeam Logoby WimVD » Fri Apr 03, 2015 9:43 am

SyNtAxx wrote:Case #00828648 Open just short of a month so far. No resolve. Very large vms are a fail on restore every time. Has anybody successfully restored a 900gb+ vm successfully on version 8.0 Patch1 using any transport method? I've run into just about every known bug so far, requiring hotfixes to fix hotfixes. We are fortunate to have a very capable collection of hardware, but not sure why I am running into so many issues. I can admit to not being fully aware of all the nuances of the software features. However, at the core the configuration is good.

Environment:

Veeam Backup manager: vm @ 8 core 16gb ram and no proxy services, strictly a job manager.
proxy1: HP DL 580G7 4 procs @ 6 core + HT and 32gb ram. 4gb fiber hba to fabric. Teamed 1gE @ LACP. Win2k12R2
proxy2: HP BL460c g8: 2 procs @ 10 core + HT and 512gb ram and 2 @ 10gE for network and 2 @ 8gbps FC HBA. Win2k12R2 Blade chassis is 8x8gbps direct connected to SAN Frame and 40gE to the core network.

SAN:

3par 10800 fully loaded.
8 controllers
1700 drives.


Nick


I have restored 1TB+ disks using Instant VM restore and storage vMotion. Other methods failed for different reasons as you mention.
Looking at your hardware it worries me that you only restore at 90MB/s. At that rate restoring big disks just takes too long.
Granted you can use instant restore to bring the VM online while restoring but what if you only need to restore a single big disk.
A very common scenario with cryptolocker viruses these days...

I hope Veeam will make restore speed and stability a priority in the next patches/releases. But I also understand that they are very dependent on VMware in this case.
There is a very good possibility this is not (only) a Veeam issue but VMware as well.
Knowing Veeam i'm sure it will get resolved.
WimVD
Service Provider
 
Posts: 48
Liked: 10 times
Joined: Tue Dec 23, 2014 4:04 pm

Re: Incredible slow Direc SAN restore

Veeam Logoby SyNtAxx » Tue Apr 07, 2015 9:31 pm

With such large drives becoming fairly common things should work. I hesitate to use the instant vm recovery (have done it a few time with success) but I'm not sure how it will work for a High IO server. I tried redirecting the change file to a san, but to be honest I don't recall the outcome of that test. Be that as it may, moving a very large vm to san via NFS can take quite a bit of time.

What would be nice to see and I've requested, is the ability to schedule automatic san snapshots via the storage integration/san explorer (if you l have it licensed) and use those as recovery points. I know they talk about it in the user manual being a very fast way to recover, and indeed it is as I've recovered a test server in about 3 minutes. I'm still learning the finer details of the application, but I don't think you can auto schedule san snaps to use as recovery points currently. They are manual at this point. That feature would be awesome for tier1 applications. I can do a lot of that with an application I have called HP Recovery Manager, but would be nice to have all function in one recovery app.

Nick
SyNtAxx
Expert
 
Posts: 127
Liked: 14 times
Joined: Fri Jan 02, 2015 7:12 pm

Re: Incredible slow Direc SAN restore

Veeam Logoby WimVD » Fri Apr 10, 2015 12:11 pm

SyNtAxx wrote:With such large drives becoming fairly common things should work. I hesitate to use the instant vm recovery (have done it a few time with success) but I'm not sure how it will work for a High IO server. I tried redirecting the change file to a san, but to be honest I don't recall the outcome of that test.


Remember you don't need to bring the server online when using instant restore. You can just publish it and move the VM or a disk offline.
WimVD
Service Provider
 
Posts: 48
Liked: 10 times
Joined: Tue Dec 23, 2014 4:04 pm

Re: Incredible slow Direc SAN restore

Veeam Logoby Delo123 » Mon Apr 13, 2015 9:26 am

I remeber using instant restore one time after a bad bad san crash. During vmotion the instant recovery process stopped due to a decompression error if i remember correctly. Backup was on a windows deduped volume.
Maybe some thing were not fully in order, however what scared me the most is that the instant recovery job showed 5 retries (but ofcourse they also failed) and then shut down the vm. Since this was our only "live" recovery exchange database we actually had some dataloss, because after the vm shutdown all changes were gone and there is no option to reboot (to recover other data then the 1 corrupt block) so i guess i will never again use it for production restore where changes are important
Delo123
Expert
 
Posts: 351
Liked: 97 times
Joined: Fri Dec 28, 2012 5:20 pm
Full Name: Guido Meijers

Re: Incredible slow Direc SAN restore

Veeam Logoby foggy » Tue Apr 21, 2015 1:30 pm

Guido, have you contacted support to investigate the reasons of this behavior?
foggy
Veeam Software
 
Posts: 14752
Liked: 1083 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Incredible slow Direc SAN restore

Veeam Logoby Delo123 » Tue Apr 21, 2015 2:29 pm

Actually we didn't. You can imagine it was quite a hard weekend, and in the end we knew something went wrong on the windows deduplication target...
We did find the error in the Veeam KB's (or Forums) confirming there was a read "error" from the repository. However this doesn't really help.
For us we learned from this is that Surebackup / Instant Recovery cannot be reliable enough unless it actually reads all of the data within the backup file which of course is impossible to do practically.
That is where replica's come into play of course....

I thought about opening a ticket later on to discuss the matter of actually keeping the changed data after an instant recovery failes (or skips the bad block) without actually taking the vm down.
Actually i tried finding someboby at Vmworld to talk to regarding this and everybody at the booth pointed to Anton to discuss this. When i finally met him briefly at the veeam party (greatest party at the show as always) other things were more important :):) and i actually forgot about it.... But maybe it would be a good thing to discuss internally for you guys.
Delo123
Expert
 
Posts: 351
Liked: 97 times
Joined: Fri Dec 28, 2012 5:20 pm
Full Name: Guido Meijers

Re: Incredible slow Direc SAN restore

Veeam Logoby SyNtAxx » Wed Apr 22, 2015 3:35 pm

I have an update on my failed SAN restores: We uncovered a bug in the SAN recovery method. I custom fix was created that will be released post update 2.

I still have slow SAN restores speeds. Restoring to a local 5 disk raid 5 array is 2x faster (200=MB/sec) then the 1700 drive array 89-120MB/sec). Local drive has no VAAI integration, I've tried disabling VAAI on a test esx host but yielded no improvement in speed. One thing I hadn't tried yet is created a new lun exported only to the test esxi server and disabling VAAI again.

-Nick
SyNtAxx
Expert
 
Posts: 127
Liked: 14 times
Joined: Fri Jan 02, 2015 7:12 pm

PreviousNext

Return to VMware vSphere



Who is online

Users browsing this forum: Google [Bot], Richie.B and 20 guests