Ever had a VM that NEVER chooses SAN mode transport?

[nickl99]

Our exchange server is doing this. Every other server on the same LUNS, at the same site using the same backup proxy handles SAN mode transport as expected, so I know the local repository has valid paths to all the disk stores for this VM. For reference all up it has 7 disks, all thick provisioned, spread over 2 different FC connected stores. 4 TB all up, the biggest disk is 1TB (only 10% used).

I have only noticed this in the last few months and was hoping it would fix itself after some reboots, updates etc. but No even after recently Upgrading all our esx hosts to 6.0.0, 3620759 and full shuts of all Virtual environment, Veaam always fails to use SAN mode when going through the transport options, and ends up falling back to Hotadd over network, rather than superior FC.

I tried to get Veeam support involved and they tried investigating it but all they could show is that *it should work* "but for some reason VMWare is rejecting its API calls for accessing the disks over SAN mode, but fails and falls back to network. The logs reporting this werent very convincing, I cant remember exactly at the time, but it was something like "access denied" which doesnt quite make sense as FC is a media based connection and apart from zoning and read\write permissions at the brocade level (which is fine for all the disks attached, and can be shown works for the other jobs).

So Veeam support passed it off as a VMWare problem, and told me to contact them. Has anyone been through a similar issue with certain VM's in certain states? (possibly orphaned snapshots or something? ive looked on the store but cant find such).

Should I go ahead and log this with VMware and how should I go about explaining the problem to them in a way that they feel inclined to help (and take ownership?).

Thanks

[foggy]

For reference all up it has 7 disks, all thick provisioned, spread over 2 different FC connected stores. 4 TB all up, the biggest disk is 1TB (only 10% used).

Are you positive that direct SAN works for other VMs having disks stored on the very same LUNs? There're some limitations of using direct SAN, but it doesn't look as any of them apply. Anyway, the particular reason of the failure is mentioned in the job debug log and is the one that should be investigated with the help of support. VMware could check logs on their side for the actual reason of rejecting the API call.

[nickl99]

Positive. Other jobs with disks on the same stores, are able to use SAN mode without a problem.

This all the Veeam log says, nothing that makes sense. I think i recall when the Veeam tech explained it to me: the access rights error means Veeam was denied access via SAN mode when making the call to VMWare API

[PROD_TIER01_VOL_0] clstexchsvr01/clstexchsvr01_2.vmdk.

[28.07.2016 07:53:33] < 21584> dsk| WARN|VDDK logon has failed. Logon specification: [VC/ESX: [vsphere.domain.local];Port: 443;Login: [Domain\sac_VeeamBackup];VMX Spec: [moref=vm-165];Snapshot mor: [snapshot-45391];Transports: [san];Read Only: [true]]. VMDK path: [[PROD_TIER01_VOL_0] clstexchsvr01/clstexchsvr01_2.vmdk]. Read only mode: [true].
[28.07.2016 07:53:33] < 21584> dsk| >> |VDDK error: 13 (You do not have access rights to this file). Value: 0x000000000000000d
[28.07.2016 07:53:33] < 21584> dsk| >> |Failed to open virtual disk [PROD_TIER01_VOL_0] clstexchsvr01/clstexchsvr01_2.vmdk (flags: 4)

[veremin]

Then, it's high time to reach VMware support team and ask for assistance. Thanks.

[foggy]

Right, for some reason, VDDK considers you do not have access to this particular VMDK. Most likely VMware will be able to define what's so special about this disk file.

[nickl99]

BUMP

It turns out there are many many more VM's that arent able to use SAN mode. So its looking like it could be Veeam proxy related. How can i systematically check that the Veeam proxy/repository has a valid non-LAN path to stores for all the VM's?

The weird thing is the issue is only occuring at site 1 which houses most of our production servers. If you look into the setup of this repository/proxy, in 'connected datastores' it is set to manual but definetely lists all the datastores that exist at that site.
WHEREAS at site 2 it is set to 'automatic' BUT the listed connected datastores are all wrong, and not actually directly connected to that repository. It actually lists all the site1 datastores. However all jobs running through this proxy are able to use SAN mode transport.

I have just opened a Veeam ticket (01926693) to get to the bottom of this, but I thought I would bump this thread.

Ideas?

[foggy]

What if you try automatic for the connected datastores in site 1?

[nickl99]

Tried that one. No dice.

Just to update issue: I have gone through every store that site1 repository should have access to and SAN mode seems universally broken for particular stores, but not all. I have checked every job\server and its common to the datastore, results below:

Datastore SAN mode?
PROD_TIER01_VOL_0 no
PROD_TIER01_VOL_1 no
PROD_TIER01_VOL_2 no
PROD_TIER01_VOL_3 no
PROD_TIER01_VOL_4 no
PROD_TIER01_VOL_5 yes
PROD_TIER01_VOL_6 yes
PROD_TIER01_VOL_7 yes

So, what are this symptomatic of? Is there a way in disk manager to check off GUID's against the stores and check that there is indeed a path?

I am also reading about Windows native MPIO driver sometimes causing issues. Tried removing it, all of a sudden i get 8 times more disks come up in disk manager, but the same jobs\servers still fail to use SAN mode. I've re-installed windows MPIO.

[nickl99]

Found the issue thanks. In the end it was the SAN host mappings that were missing. I should have looked there first but a colleague got all defensive about it being the cause as he had done the initial config and sign off. Cheers all.

[Vitaliy S.]

It turns out there are many many more VM's that arent able to use SAN mode. So its looking like it could be Veeam proxy related. How can i systematically check that the Veeam proxy/repository has a valid non-LAN path to stores for all the VM's?

Maybe this assessment report will come in handy next time > Unmapped Datastore LUNs

[nickl99]

Yes that would be very useful (he asked for proof before he even logged on to look at). I cant find that report on Veeam ONE. Where can i get it and how do you load new reports to VO?

[veremin]

Reporter -> Workspace -> Veeam Backup and Replication -> Unmapped Datastore LUNs. Thanks.

[Vitaliy S.]

To find this report in the Veeam Backup & Replication report pack, first you need to make sure that Veeam backup console has been added to Veeam ONE. After that, please follow Vladimir's guidelines.