Ever had a VM that NEVER chooses SAN mode transport?

VMware specific discussions

Ever had a VM that NEVER chooses SAN mode transport?

Veeam Logoby nickl99 » Thu Oct 06, 2016 2:03 am

Our exchange server is doing this. Every other server on the same LUNS, at the same site using the same backup proxy handles SAN mode transport as expected, so I know the local repository has valid paths to all the disk stores for this VM. For reference all up it has 7 disks, all thick provisioned, spread over 2 different FC connected stores. 4 TB all up, the biggest disk is 1TB (only 10% used).

I have only noticed this in the last few months and was hoping it would fix itself after some reboots, updates etc. but No even after recently Upgrading all our esx hosts to 6.0.0, 3620759 and full shuts of all Virtual environment, Veaam always fails to use SAN mode when going through the transport options, and ends up falling back to Hotadd over network, rather than superior FC.

I tried to get Veeam support involved and they tried investigating it but all they could show is that *it should work* "but for some reason VMWare is rejecting its API calls for accessing the disks over SAN mode, but fails and falls back to network. The logs reporting this werent very convincing, I cant remember exactly at the time, but it was something like "access denied" which doesnt quite make sense as FC is a media based connection and apart from zoning and read\write permissions at the brocade level (which is fine for all the disks attached, and can be shown works for the other jobs).

So Veeam support passed it off as a VMWare problem, and told me to contact them. Has anyone been through a similar issue with certain VM's in certain states? (possibly orphaned snapshots or something? ive looked on the store but cant find such).

Should I go ahead and log this with VMware and how should I go about explaining the problem to them in a way that they feel inclined to help (and take ownership?).

Thanks
nickl99
Enthusiast
 
Posts: 26
Liked: 2 times
Joined: Wed Aug 03, 2016 5:09 am
Full Name: NICK LAING

Re: Ever had a VM that NEVER chooses SAN mode transport?

Veeam Logoby foggy » Thu Oct 06, 2016 11:23 am

nickl99 wrote:For reference all up it has 7 disks, all thick provisioned, spread over 2 different FC connected stores. 4 TB all up, the biggest disk is 1TB (only 10% used).

Are you positive that direct SAN works for other VMs having disks stored on the very same LUNs? There're some limitations of using direct SAN, but it doesn't look as any of them apply. Anyway, the particular reason of the failure is mentioned in the job debug log and is the one that should be investigated with the help of support. VMware could check logs on their side for the actual reason of rejecting the API call.
foggy
Veeam Software
 
Posts: 14742
Liked: 1079 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Ever had a VM that NEVER chooses SAN mode transport?

Veeam Logoby nickl99 » Fri Oct 07, 2016 6:33 am

Positive. Other jobs with disks on the same stores, are able to use SAN mode without a problem.

This all the Veeam log says, nothing that makes sense. I think i recall when the Veeam tech explained it to me: the access rights error means Veeam was denied access via SAN mode when making the call to VMWare API

[PROD_TIER01_VOL_0] clstexchsvr01/clstexchsvr01_2.vmdk.

[28.07.2016 07:53:33] < 21584> dsk| WARN|VDDK logon has failed. Logon specification: [VC/ESX: [vsphere.domain.local];Port: 443;Login: [Domain\sac_VeeamBackup];VMX Spec: [moref=vm-165];Snapshot mor: [snapshot-45391];Transports: [san];Read Only: [true]]. VMDK path: [[PROD_TIER01_VOL_0] clstexchsvr01/clstexchsvr01_2.vmdk]. Read only mode: [true].
[28.07.2016 07:53:33] < 21584> dsk| >> |VDDK error: 13 (You do not have access rights to this file). Value: 0x000000000000000d
[28.07.2016 07:53:33] < 21584> dsk| >> |Failed to open virtual disk [PROD_TIER01_VOL_0] clstexchsvr01/clstexchsvr01_2.vmdk (flags: 4)
nickl99
Enthusiast
 
Posts: 26
Liked: 2 times
Joined: Wed Aug 03, 2016 5:09 am
Full Name: NICK LAING

Re: Ever had a VM that NEVER chooses SAN mode transport?

Veeam Logoby v.Eremin » Fri Oct 07, 2016 9:29 am

Then, it's high time to reach VMware support team and ask for assistance. Thanks.
v.Eremin
Veeam Software
 
Posts: 13266
Liked: 968 times
Joined: Fri Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin

Re: Ever had a VM that NEVER chooses SAN mode transport?

Veeam Logoby foggy » Fri Oct 07, 2016 10:01 am

Right, for some reason, VDDK considers you do not have access to this particular VMDK. Most likely VMware will be able to define what's so special about this disk file.
foggy
Veeam Software
 
Posts: 14742
Liked: 1079 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Ever had a VM that NEVER chooses SAN mode transport?

Veeam Logoby nickl99 » Tue Oct 11, 2016 4:18 am

BUMP

It turns out there are many many more VM's that arent able to use SAN mode. So its looking like it could be Veeam proxy related. How can i systematically check that the Veeam proxy/repository has a valid non-LAN path to stores for all the VM's?

The weird thing is the issue is only occuring at site 1 which houses most of our production servers. If you look into the setup of this repository/proxy, in 'connected datastores' it is set to manual but definetely lists all the datastores that exist at that site.
WHEREAS at site 2 it is set to 'automatic' BUT the listed connected datastores are all wrong, and not actually directly connected to that repository. It actually lists all the site1 datastores. However all jobs running through this proxy are able to use SAN mode transport.

I have just opened a Veeam ticket (01926693) to get to the bottom of this, but I thought I would bump this thread.

Ideas?
nickl99
Enthusiast
 
Posts: 26
Liked: 2 times
Joined: Wed Aug 03, 2016 5:09 am
Full Name: NICK LAING

Re: Ever had a VM that NEVER chooses SAN mode transport?

Veeam Logoby foggy » Tue Oct 11, 2016 11:16 am

What if you try automatic for the connected datastores in site 1?
foggy
Veeam Software
 
Posts: 14742
Liked: 1079 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Ever had a VM that NEVER chooses SAN mode transport?

Veeam Logoby nickl99 » Wed Oct 12, 2016 1:47 am

Tried that one. No dice.

Just to update issue: I have gone through every store that site1 repository should have access to and SAN mode seems universally broken for particular stores, but not all. I have checked every job\server and its common to the datastore, results below:

Datastore SAN mode?
PROD_TIER01_VOL_0no
PROD_TIER01_VOL_1no
PROD_TIER01_VOL_2no
PROD_TIER01_VOL_3no
PROD_TIER01_VOL_4no
PROD_TIER01_VOL_5yes
PROD_TIER01_VOL_6yes
PROD_TIER01_VOL_7yes

So, what are this symptomatic of? Is there a way in disk manager to check off GUID's against the stores and check that there is indeed a path?

I am also reading about Windows native MPIO driver sometimes causing issues. Tried removing it, all of a sudden i get 8 times more disks come up in disk manager, but the same jobs\servers still fail to use SAN mode. I've re-installed windows MPIO.
nickl99
Enthusiast
 
Posts: 26
Liked: 2 times
Joined: Wed Aug 03, 2016 5:09 am
Full Name: NICK LAING

Re: Ever had a VM that NEVER chooses SAN mode transport?

Veeam Logoby nickl99 » Wed Oct 12, 2016 6:18 am 1 person likes this post

Found the issue thanks. In the end it was the SAN host mappings that were missing. I should have looked there first but a colleague got all defensive about it being the cause as he had done the initial config and sign off. Cheers all.
nickl99
Enthusiast
 
Posts: 26
Liked: 2 times
Joined: Wed Aug 03, 2016 5:09 am
Full Name: NICK LAING

Re: Ever had a VM that NEVER chooses SAN mode transport?

Veeam Logoby Vitaliy S. » Mon Oct 17, 2016 12:35 pm 1 person likes this post

nickl99 wrote:It turns out there are many many more VM's that arent able to use SAN mode. So its looking like it could be Veeam proxy related. How can i systematically check that the Veeam proxy/repository has a valid non-LAN path to stores for all the VM's?

Maybe this assessment report will come in handy next time > Unmapped Datastore LUNs
Vitaliy S.
Veeam Software
 
Posts: 19558
Liked: 1102 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: Ever had a VM that NEVER chooses SAN mode transport?

Veeam Logoby nickl99 » Tue Oct 18, 2016 2:16 am

Yes that would be very useful (he asked for proof before he even logged on to look at). I cant find that report on Veeam ONE. Where can i get it and how do you load new reports to VO?
nickl99
Enthusiast
 
Posts: 26
Liked: 2 times
Joined: Wed Aug 03, 2016 5:09 am
Full Name: NICK LAING

Re: Ever had a VM that NEVER chooses SAN mode transport?

Veeam Logoby v.Eremin » Tue Oct 18, 2016 8:45 am

Reporter -> Workspace -> Veeam Backup and Replication -> Unmapped Datastore LUNs. Thanks.
v.Eremin
Veeam Software
 
Posts: 13266
Liked: 968 times
Joined: Fri Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin

Re: Ever had a VM that NEVER chooses SAN mode transport?

Veeam Logoby Vitaliy S. » Tue Oct 18, 2016 11:18 am

To find this report in the Veeam Backup & Replication report pack, first you need to make sure that Veeam backup console has been added to Veeam ONE. After that, please follow Vladimir's guidelines.
Vitaliy S.
Veeam Software
 
Posts: 19558
Liked: 1102 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov


Return to VMware vSphere



Who is online

Users browsing this forum: No registered users and 11 guests