Direct SAN access failure troubleshooting

[massimiliano.rizzi]

Hello experts,

Case ID 5195165

we have just finished installing and configuring Veeam Backup & Replication for VMware (build 6.0.0.181) in a new dedicated IBM x3650 physical Windows server running Windows Server 2008 x64 SP2. This server is also equipped with 2x QLogic 4Gb FC Single-port HBAs for IBM System x so that it can use Direct SAN Access to the VMFS datastores.

VMFS datastores are residing on an EMC CLARiiON AX4-5F Dual SP enclosure with Fibre Channel front-end ports running.
Our vSphere hosts are running VMware ESXi VMware vSphere 5.0 Update 1. We are also using the new vCenter Server Virtual Appliance (vCSA).

In order to setup Direct SAN Access to the VMFS datastores in the proper way, we have set the following configurations:

1) SAN visibility: the backup server where the Veeam Backup 6 environment is installed does have access to the VMFS datastores.
2) Disable Automount: we have disabled the automount feature of Windows to aviod disk re-signature using diskpart utility.
3) Compatibility: Veaam Backup 6 has been upgraded to build 6.0.0.181.
4) Multipath: the Windows Server 2008 x64 SP2 Server where the Veeam Backup 6 environment is installed has Native Multipath installed and configured using the Failover Only load-balancing policy setting.

When the backup proxy uses default settings with regards to data transport modes (Direct SAN Access > Virtual Appliance > Network) Veeam Backup & Replication automatically chooses Network transport mode.

==================================================
Unable to establish direct connection to the shared storage (SAN)
==================================================

If we change the backup proxy configuration to use Direct SAN Access transport mode only and disable the “Failover to network mode if primary transport modes fail or are unavailable” option SAN Backups immediately fail with the following error:

==================================================
Unable to allocate processing resources. Error: No backup proxy is able to backup this VM. Check processing mode settings on proxies.
==================================================

Any ideas on how to start troubleshooting the problem will be greatly appreciated.

Thanks and Regards,

Massimiliano

[Gostev]

Hi, this means Veeam does not see the required VMFS LUNs attached to any of its backup proxy servers.
- Do you see VMFS volumes in the Disk Management snap-in on the Veeam backup proxy server?
- If you attached VMFS volumes after installing the product, did you perform storage rescan in Veeam? This happens automatically every 24 hours, too.
- Try selecting your VMFS datastores manually in the Veeam backup proxy server properties. This helps when VMFS LUN cannot be detected for whatever reason.
- Some multipathing software is known to cause issues, so the final troubleshooting step might require removing that.
Thanks!

[habibalby]

Your environment same as mine, EMC AX4 but iSCSI not FC.

I agree entirely with @Gostev, some MPIO if not all of them creates a problem to a access the VMFS LUNs in Windows. In my environment, it was working fine, as soon as I introduced the MPIO of IBM DS3500, the jobs started to give warning about Direct SAN Access.

Allow Veeam Server to access the VMFS LUNs using only one path/IP of the controller and try Direct SAN Access.

Thanks,

[habibalby]

Hello,
As I told you, after disabling the MPIO the SAN Mode works perfectly.

vCenter Tasks:


Veeam Job:

[massimiliano.rizzi]

Hi Gostev/habibalby,

thank you for your replies.

I completely agree with Gostev, Veeam did not properly see the required VMFS LUNs attached to our backup proxy server while I was able to see the VMFS volumes as non-initialized LUNs without drive letters in disk management console snap-in on the Veeam backup proxy server.

VMFS volumes have been attached after installing the product, however I had already performed a manual storage rescan in Veeam.

I have also checked Veeam KB article ID 1446 and apparently everything should be ok with regard to SAN access setup for use with Veeam Backup & Replication.

What I did after realizing that Veeam was not properly seeing the required VMFS LUNs attached to our backup proxy server is installing VMware VCB framework 1.5 Update 2 for vSphere 4.1 on our Veeam Backup server and running vcbSanDbg.exe diagnostic tool included. After investigating its output I realized (as Gostev said) that it was not able to see any LUNs containing VMFS signatures (I know that VCB components do not support VMFS5, however we are still on VMFS3).

Since we are using EMC storage, I managed to install the latest version of PowerPath with service packs (PowerPath 5.5 SP1 for Windows X64 Bit) after removing Windows MPIO and rebooting our Veeam Backup server.

As for the latest output of vcbSanDbg.exe after installing PowerPath, it can see 2 VMFS signatures as expected since we currently have 2 VMFS datastores:

Code: Select all

 [2012-06-04 12:01:16.491 'App' 5496 info] Found SCSI Device: NAA:60060160aed02500600c4164c296de11524149442035
 [2012-06-04 12:01:16.491 'App' 5496 info] Visible on 1 paths:
 [2012-06-04 12:01:16.491 'App' 5496 info] Device Name: \\?\scsi#disk&ven_emc&prod_power&#{4a54205a-c920-4e28-88c5-9a6296a74b0b}&emcp&power9#{53f56307-b6bf-11d0-94f2-00a0c91efb8b}, Bus: 0 Target: 0 Lun: 4
 [2012-06-04 12:01:16.491 'App' 5496 info] Lun contains VMFS/LVM signatures:
 [2012-06-04 12:01:16.491 'App' 5496 info] VMFS LVM signature.
 [2012-06-04 12:01:16.491 'App' 5496 info] ID: LVID:4a9d4c99-adb51a6a-1b04-00215e09e8ea/4a9d4c99-9175504c-8e99-00215e09e8ea/1 Name: 4a9d4c99-9175504c-8e99-00215e09e8ea
 [2012-06-04 12:01:16.491 'App' 5496 info] Found SCSI Device: NAA:60060160aed02500e8ba7a003ed1df11524149442031
 [2012-06-04 12:01:16.491 'App' 5496 info] Visible on 1 paths:
 [2012-06-04 12:01:16.491 'App' 5496 info] Device Name: \\?\scsi#disk&ven_emc&prod_power&#{4a54205a-c920-4e28-88c5-9a6296a74b0b}&emcp&power17#{53f56307-b6bf-11d0-94f2-00a0c91efb8b}, Bus: 0 Target: 0 Lun: 8
 [2012-06-04 12:01:16.491 'App' 5496 info] Lun contains VMFS/LVM signatures:
 [2012-06-04 12:01:16.491 'App' 5496 info] VMFS LVM signature.
 [2012-06-04 12:01:16.491 'App' 5496 info] ID: LVID:4cac96c7-154b469b-8aba-001517bc6a9c/4cac96c6-f8872a15-66f1-001517bc6a9c/1 Name: 4cac96c6-f8872a15-66f1-001517bc6a9c

As for vcbSanDbg.exe output, the remaining LUNs that does not contain any VMFS/LVM signatures refer to local storage of the physical Windows server running Veeam and to storage LUNs directly connected from the storage area network to a couple of virtual machines as RDM configured in virtual compatibility mode.

How can we further troubleshoot the problem ?

Thanks and Regards,

Massimiliano

[Gostev]

Did you remember to do storage rescan in Veeam again after getting done with MPIO stuff?
If yes, did you also try bullet 3 in my original response?

[habibalby]

Can you focus on one VMFS LUN that presented to the backup proxy and one VM inside that LUN?

[habibalby]

remove all the mapping in the proxy, reboot and make sure no LUNS aee seen in the disk management. Present one vmfs lun, reboot. Make sure you see the lun in the disk management, reboot, piCk only one VM inside that LUN, force the job to take the proxy where the lun is presented, make sure the proxy configuration as san mode.

Are you still getting the warning?

[massimiliano.rizzi]

Hi Gostev/habibalby,

thank you for writing further.

I have finally been able to use Direct SAN Access transport mode successfully after trying bullet 3 in the below response from Gostev:

Hi, this means Veeam does not see the required VMFS LUNs attached to any of its backup proxy servers.
- Do you see VMFS volumes in the Disk Management snap-in on the Veeam backup proxy server?
- If you attached VMFS volumes after installing the product, did you perform storage rescan in Veeam? This happens automatically every 24 hours, too.
- Try selecting your VMFS datastores manually in the Veeam backup proxy server properties. This helps when VMFS LUN cannot be detected for whatever reason.
- Some multipathing software is known to cause issues, so the final troubleshooting step might require removing that.
Thanks!

Selecting the VMFS datastores manually in the Veeam backup proxy server properties did help us.

Hopefully after getting back to this Veeam backup proxy server tomorrow I will perform an in-place upgrade to the latest 6.1 build, revert changes to select the VMFS datastores automatically as for the default settings in the Veeam backup proxy server properties and see if that helps.

Since our Veeam backup proxy server also has Symantec Backup Exec 2012 with the agent for virtualization, I also had the opportunity to make a performance/functionality comparison between Symantec Backup Exec 2012 with the agent for virtualization and Veeam and I confirm there is no comparison, Veeam really rocks.

We will definitely switch our customers using Symantec Backup Exec 2012 with the agent for virtualization to Veeam resulting in Symantec Backup Exec 2012 being only used to duplicate Veeam backup files to tape media for off-site archive.

Thank you again for your support.

Regards,

Massimiliano

[habibalby]

Glad to hear that its working, what were the steps taken to make it works, perhaps somone else benifit from.

[Gostev]

Massimo, thank you for your kind words. We are working to remove that final reason for your customers to still use Symantec in addition to Veeam

[massimiliano.rizzi]

Glad to hear that its working, what were the steps taken to make it works, perhaps somone else benifit from.

Hi there,

glad to share the fix.

Basically, I opened the settings of our backup proxy server and switched from automatic (recommended) datastore selection option to manual datastore selection so I had to manually select the datastores where our VMs reside.

Regards,
Massimiliano

[LeoJBI]

Hello,

I am looking into an error I am receiving:

Unable to establish direct connection to the shared storage (SAN)

My question right now is the suggestion I found within the knowledgebase states to install the VCB framework. I am looking for the vcbsandbg.exe tool within that framework. However, I do not know where to find these tools.

I intend to run the diagnostic but I am not certain this is my problem. I can see the LUN within the Backup Repositories, I can see it within the Windows Disk Management. Therefore I am not certain what my problem is?

Leo

[LeoJBI]

Thanks...I guess?

To start, I simply wanted to find the VCB framework (tools?) in order to run the diagnostic (using the vcbsandbg.exe). Instead, I get redirected to a long tread on how someone solved his problem by manually selecting his datastore within the proxy server. Great!

However, In my case, it already is manually selected…

[Gostev]

I don't know if it is clear enough above, but VCB does not "understand" VMFS5, so this way of troubleshooting is no longer an option with vSphere 5. Please go through all the troubleshooting steps above carefully. If nothing helps, then you should open a support case (since the above list is exhaustive).

[cffit]

This is in reference to support case 5190339.

I have an EMC Celerra NX4 iscsi SAN that I am trying to do Direct SAN backups with VEEAM. I have this working fine with our Equallogic iscsi SAN, so I'm familiar with how this is done. I can get the datastore volumes on the EMC SAN to show up on the VEEAM backup server the same way I see the Equallogic ones, but when I attempt to do backup I can only do network mode. SAN mode fail every time. We just did a flare code upgrade of the EMC SAN this last weekend and I thought that might resolve the issue as I had read for native Windows MPIO we needed to be at a certain level of flare code on our SAN. Now we are past that version, but still the VEEAM direct san backups fail.

Has anyone else had issues with Direct SAN backup using EMC, specifically Celerra/Clariion models? Do you have to use PowerPath even though you can get the volumes to show up properly in Disk Manager on the VEEAM backup server?

[kocher]

Hi,
we are using FC and ISCSI. Since upgrading to B&R 6.5 san mode was failing.
We were able to get it to work by selecting datastores manually in the proxy configuration.
Switching back to Automatic causes backup to fail again.
Is there anyway to get the proxy to detect datastores automatically again or are we condemned to add every new datastore manually?

Thanks
Kristian

[habibalby]

Hello @Kocher,

I was planning to postponed the installation of 6.5 next week after the SAN Migration, but I saw your post and I said let me try it out if I can generate any errors.

I have installed 6.5 on a test VM and set it with 3 vNICs, 1vNIC for Production, 1vNIC for vSphere Management Network and 1 vNIC for iSCSI Initiator to connect to the iSCSI Fabric.

I have configured one Job for a "Test VM" on same ESX host where the "Veeam Test VM" running and Mapped one LUN where the "Test VM" resides to the "Veeam Test VM". I configured the Proxy to be as Direct SAN Access and disabled the Fail-over to Network mode Option...

It detected the SAN Mode and backup went fine without any issue.
I configured the Job to be a Virtual Appliance and the backup went fine without any issue.

I you want to backup your VMs via SAN Mode, just make sure your Proxy Servers can see the VMFS LUNs inside the Disk Management and in the proxy configuration, select the backup mode as Direct SAN... Then in the job, make sure you are selecting the Proxy Server which has access to the iSCSI SAN, "VMFS LUNs"...

[habibalby]

@Kocher,
Are you seeing this error ??

Code: Select all

10/31/2012 1:42:35 PM :: Unable to allocate processing resources. Error: No backup proxy is able to backup this VM. Check processing mode settings on proxies.

[habibalby]

To sort out this issue;
1. Make sure your Proxy Server can see the VMFS LUNs inside the Disk Management of the Windows.
2. Make sure the Proxy configured as Direct SAN Access.
3. Make sure the Direct SAN Access Proxy is selected in the Backup Job Configuration / Storage Section.
4. If you are still getting the error and failing to process the backup further, Open the Proxy Configuration under Backup Infrastructure -> Backup Proxies and select the desire proxy.
Right Click on the Desired Proxy and Select Properties in the Selected DataStores, select the VMFS LUNs which contains the target Backup VMs manually and try..

[kocher]

Hi,

thanks for your answers.
This is actually exactly what I'm doing.
When I select the datastores manually in the proxy configuration everything if fine.
So the proxy can communicate with the datastores just fine, but I would prefer to make this happen automatically so I don't have to add new datastores manually on all the proxies...

Kristian

[habibalby]

Hi,
With Automatic Selection there is an issue for quite sometime.. I think Veeam should come up with a result..

Have you tried assigning the LUN to only One Controller as a Preferred Controller and the Proxy Server connected ONLY to that Controller and not to the other and set the DataStore as Automatic Selection.. May be that will work and the proxy connection will go directly to the preferred path via the active vNIC to that controller..

I don't have this feature in my SAN AX4-5 otherwise I would test it..

Thanks,

[habibalby]

Kristian

Why don't you stick to HotAdd Mode? If your Proxies are Virtual and you set them as HotAdd Mode, you will see the performance of backup rates above 100MB/s as Full and incremental 10x faster..

Thanks,

[cronosinternet]

For what it is worth...

We have an EMC VNX connected via 10Gb iSCSI. Our Veeam server is a standalone Windows 2008 R2 server with 2 x 10Gb ports. I had "lots of fun" getting Server 2008 to multipath iscsi to the EMC and only see the LUNS once in Disk Manager (requires the MPIO feature or PowerPath) but this works when done.

At which point Veeam6.5 has worked faultlessly. The iSCSI traffic is perfectly balances across both NICs.

[kocher]

The proxies are physical machines with dedicated storage so HotAdd is not the best solution.
Connecting the proxies to only one controller is not really an option because HA and redundancy are very important for this environment.

Anyway if it is a known problem and a solution will be provided (hopefully) I can live with it.

So is this issue new with 6.5 or is it just a coincidence in my case?

Kristian

[habibalby]

I've been facing this issue since Version 6.

[ashrid]

Hi,

I've got a ticket logged with support (00165086) but it doesnt seem to be progressing very quickly so I thought I'd post here in case someone else can help.

I've had Veeam v5 running for over 12 months now without issue under Wind 2008 R2. Last week I had to upgrade the disk array that Veeam backs up to as it was too small and since then my Direct SAN access is failing. Network access is not an option as the job takes over 70 hours.

Initially support advised me to upgrade to 6.5 but this has not resolved the issue (even with the proxy configured) and now they are just referring me to the iSCSI configuration kb article http://www.veeam.com/kb_articles.html/K ... %20failure

I have recreated the iSCSI initiators several times (no changes have ben made to the EqualLogic PS6000 SAN) and I believe the issue is due to the fact that the volumes are showing up in Windows disk managment as Unallocated (pretty sure they were unknown previously) but I've not been able to find any way to change this (I have a very health fear or messing with this as my predecessor lost his job for mounting these and corrupting them) in order to test this theory.

I've seen reference to multipath causing this problem (http://www.petri.co.il/how-to-deal-with ... indows.htm) but my backup server only has a single NIC on the SAN side so I don't have multipath installed

Would appreciate any pointers

Thanks

[Cokovic]

Go to the properties of your backup proxy and try to switch the detection of your datastores from automatic to manual and add them there. See if it helps.

[ashrid]

Go to the properties of your backup proxy and try to switch the detection of your datastores from automatic to manual and add them there. See if it helps.

Thanks for the suggestion, unfortunately that has been tried and didn't make any difference

[habibalby]

Hi, try with test VMFS3 with one VM on it and this LUN presented to one host, keep the proxy running on Host and the VM inside the LUN on the same host as well. Bring the VMFS disk online, don't initiate it as it will format The VMFs store . try the proxy as direct San without network fail over. If it fails upgrade the VMG to VMFs5 and try, if it fails, add the data store manually, if it fails, add the host to Veeam Console select the teat VM and add the data store manually to the proxy from the host itself.