veeam 7 3par storage snapshot backups

[emachabert]

Case #00461892.

[mconte]

Case #00470047

Backup fails with this error: Unable to find volume with node WWN XXXXXXXXXXXXXXXX and LUN ID 1.
"XXXXXXXXXXXXXXXX" is the 3Par node WWN. LUN number is correct too.

I see in 3Par management console event log that veeam 3Par user issued those commands:

Code: Select all

createsv -comment {Do not delete! etc etc} Veeam_VOLUMENAME  VOLUMENAME
createvlun Veeam_VOLUMENAME 1 BACKUPPROXY
(after 1-2 minutes)
removevv Veeam_VOLUMENAME
removevlun Veeam_VOLUMENAME 1 BACKUPPROXY

Everything seems to be correct, snapshot and vlun are created properly.

Direct SAN backups without snapshot integration work flawlessy, so I don't think that it's a fabric related problem.

[Gostev]

Do you use QLogic HBA by any chance?

[mconte]

Do you use QLogic HBA by any chance?

Yes, a rebranded (HP FC1142SR) Qlogic QLE2460 4Gb.

[Gostev]

We are actively working on this issue right now. QLogic drivers do not seem to fully implement specifications, resulting in empty LUN mapping information returned to the corresponding WMI query. This can be reproduced without Veeam involved, so we have opened support cases with both HP and QLogic.

[chjones]

We also use qlogic hbas in our HP Blades, good to know you have found the likely issue.

For the moment we're just running direct-San without issue. We have a few VMs that are a few TB in size and it takes a long time to remove their snapshots following a backup so will be great when this is resolved and the snapshots only exist for a few minutes.

Please keep us posted on any developments.

Thanks!!!!

[mconte]

I just noticed that the same problem affects Veeam explorer for storage snapshots too ("Create snapshots" and "rescan volume").

[emachabert]

Anton, do you have any update on this issue ?

[Gostev]

No news, the case is super slow... we are escalating it, but if you can push HP from your side too, this won't hurt. The HP support case ID is 4713925829. We will be opening a couple more cases with HP on behalf of two other clients today.

[phokay]

Hi All,

Good news. I had the same problem and I looked at the issue very closely at various levels: vCenter, ESXi, Windows (Veeam Proxy), 3Par storage, etc. etc. I found that the LUN ID in question was indeed the LUN ID for the storage snapshot for the job.
The problem was that when multiple backup jobs run concurrently, in my test lab where I had only one datastores and a few VMs, Veeam tries to take multiple snapshots of the same volume concurrently as well. Guess what? the jobs sometimes failed to delete the storage snapshots (probably because of a timing issue) due to snapshots mix up on the 3Par storage array. So the workaround is to delete the storage snapshots created by Veeam on 3Par while all backup jobs are NOT running on that particular volume(s). And try to rerun the jobs. Sometimes, you will need to run full backup rather than incremental. Sometime, you will have to recreate brand new job for the VM. It works for me at the moment. When you get backups from storage snapshots working again, it's very fast and you'll be happy again.

Hopefully Veeam and HP will find the real cause and the solution very soon.

-Mike

[Gostev]

The real cause of the issue discussed in this thread is well known. It is QLogic HBA driver issue which causes GetFcpTargetMapping WMI method to return an empty list. The same happens when trying to obtain a list by using fcinfo /mapping command.

Your issue is worth separate investigation, as I don't recall anything like this reported before.

[phokay]

Hi Anton,
My post was related to this error: "Backup fails with this error: Unable to find volume with node WWN XXXXXXXXXXXXXXXX and LUN ID x." as I was having the same problem. My setup was HP blades BL460c Gen8 on c7000 enclosure with FlexFabric and 3Par 7200. The FlexFabric we have is based on Emulex FCoE 10Gb adaptor. So I guess this issue is not just related to QLogic HBA.

[phokay]

Spoke too soon...it's broken again. I installed Veeam 7 on another server which has Emulex HBA Mezzanine card (not FlexFabric) and tested again. It works without any issue. So my conclusion for now is it does not work well with HP FlexFabric FCoE adaptors.

[Gostev]

Since your issue was intermittent already, a single test on a different HBA cannot prove anything. It would be best to open a support case for proper investigation.

[kte]

what i also have is that i can't imprt the 3 par with his dns name , only by ip
if i use the dns name he the 3par doesn't allow is.

[chjones]

I have the exact same issue when adding the 3PAR.

We have two 3PAR 7400 4-Nodes and I added the first one fine by name, but the second I can only add via IP. I've confirmed name resolution is working just fine. Its a very strange issue. I haven't opened a support case yet but may do so just to find out what the issue is.

[veremin]

Might be worth opening a ticket, indeed, as it allows us to collect as much information regarding the issue as possible. Thanks.

[chjones]

I have opened a ticked, 00476367, for this issue. I'll post back when the cause and fix is known.

[chrisdearden]

I have a customer with a similar issue - certain jobs fail when using storage snap integration. The WWN name shown when it fails is not a port name on the fabric , but the Node Name ( as against the port name).
Is it worth trying FCINFO to see if I can replicate the issue without Veeam. The Customer is using Qlogic HBA.

Case Ref: 00478986

[Gostev]

Since hardware vendors are usually very slow to react to software issues, we have decided to create a hotfix for R2 that changes LUN matching logic to use SCSI Unique Id. The hotfix is currently in testing, anyone who has a support case open on this issue will get it from support once the testing is completed (if no issues are found with this approach, of course).

[chjones]

My support case about adding my second 3PAR via name has been resolved, and it still doesn't make sense why it now works.

I had created a credential in Veeam that could connect to the 3PAR (using an AD account that has full access to the 3PAR). This wasn't working. I added the 3PAR using the default built-in admin account and it worked. After it was added I switched the credential back to the Veeam specific account and it is still working.

It makes no sense why that worked, but I'll take the win.

[chrisdearden]

Since hardware vendors are usually very slow to react to software issues, we have decided to create a hotfix for R2 that changes LUN matching logic to use SCSI Unique Id. The hotfix is currently in testing, anyone who has a support case open on this issue will get it from support once the testing is completed (if no issues are found with this approach, of course).

This is Why Veeam R&D Rocks

[emachabert]

As soon as I get that hotfix I will post our feedback.
Thanks Anton.

[emachabert]

Hotfix tested.

First results :
- storage snapshots integration is now working for backup and replication jobs
- when creating a storage snapshot from veeam in order to use the snapshot explorer feature it is not yet working

I will do further tests on monday to see clearly what is working and what is not.


Thanks.

[emachabert]

We tested again this morning.
Everything is working as expected.

Eric.

[chjones]

I got the hotfix as well from support and can confirm the 3PAR Snapshots are working for me too. FANTASTIC!!!!!

We have 6 backup jobs all with VMs on the same 3PAR Virtual Volume ... now to see how nice this plays with multiple backups and snapshots at the same time

Great work Veeam!!!!!

[Gostev]

Thanks all for updating this topic with the results.

[resruss]

Hi all,

We're currently testing SAN snapshots with Veeam 7 and have a question about the logic.

Let's say we've got several VMs in a folder spread across several datastores on the SAN and Veeam is targeting that VM folder
How does Veeam handle the snapshots?
Does it keep a SAN snapshot open if it knows there's another VM on the same datastore, or does it create a SAN snapshot for one VM, remove the snapshot and move onto the next VM, as it works as a VM level?
What impact does parallel processing have on VMs in a job on the same SAN datastore?
What about different SAN datastores?

Are there any risks / gotcha / best practices we should keep in mind when utilizing the SAN snapshots, especially with parallel processing in mind?

Thanks, it looks like great technology, we just want to make sure we're using it correctly and minimizin any possible risk or performance implications.

[foggy]

Please review the beginning of this topic, should answer at least some of your questions (as well as the corresponding user guide section).

Basically, a single SAN snapshot is created for all VMs stored on it. Different LUNs are snapshotted independently.

[chjones]

Well, I applied the Version 7.0 R2a update and it breaks the snapshots again with the same error "Unable to find volume with Node WWN ################# and LUN ID #". I've attempted to replace the DLLs that were provided by Veeam Support however the Proxies all show as "out of date" in the console and the console reports that the version of the console is too old. I guess we need updated DLLs.

I've logged a support case, 00499562, and hopefully there is an update available.