Discussions specific to the VMware vSphere hypervisor
emachabert
Veeam Vanguard
Posts: 373
Liked: 165 times
Joined: Nov 17, 2010 11:42 am
Full Name: Eric Machabert
Location: France
Contact:

Re: veeam 7 3par storage snapshot backups

Post by emachabert »

Case #00461892.
Veeamizing your IT since 2009/ Vanguard 2015,2016,2017,2018,2019

mconte
Service Provider
Posts: 9
Liked: 3 times
Joined: Oct 18, 2013 7:51 am
Full Name: Michele
Contact:

Re: veeam 7 3par storage snapshot backups

Post by mconte »

Case #00470047

Backup fails with this error: Unable to find volume with node WWN XXXXXXXXXXXXXXXX and LUN ID 1.
"XXXXXXXXXXXXXXXX" is the 3Par node WWN. LUN number is correct too.

I see in 3Par management console event log that veeam 3Par user issued those commands:

Code: Select all

createsv -comment {Do not delete! etc etc} Veeam_VOLUMENAME  VOLUMENAME
createvlun Veeam_VOLUMENAME 1 BACKUPPROXY
(after 1-2 minutes)
removevv Veeam_VOLUMENAME
removevlun Veeam_VOLUMENAME 1 BACKUPPROXY
Everything seems to be correct, snapshot and vlun are created properly.

Direct SAN backups without snapshot integration work flawlessy, so I don't think that it's a fabric related problem.

Gostev
SVP, Product Management
Posts: 26524
Liked: 4155 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: veeam 7 3par storage snapshot backups

Post by Gostev »

Do you use QLogic HBA by any chance?

mconte
Service Provider
Posts: 9
Liked: 3 times
Joined: Oct 18, 2013 7:51 am
Full Name: Michele
Contact:

Re: veeam 7 3par storage snapshot backups

Post by mconte »

Gostev wrote:Do you use QLogic HBA by any chance?
Yes, a rebranded (HP FC1142SR) Qlogic QLE2460 4Gb.

Gostev
SVP, Product Management
Posts: 26524
Liked: 4155 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: veeam 7 3par storage snapshot backups

Post by Gostev »

We are actively working on this issue right now. QLogic drivers do not seem to fully implement specifications, resulting in empty LUN mapping information returned to the corresponding WMI query. This can be reproduced without Veeam involved, so we have opened support cases with both HP and QLogic.

chjones
Expert
Posts: 108
Liked: 29 times
Joined: Oct 30, 2012 7:53 pm
Full Name: Chris Jones
Contact:

Re: veeam 7 3par storage snapshot backups

Post by chjones »

We also use qlogic hbas in our HP Blades, good to know you have found the likely issue.

For the moment we're just running direct-San without issue. We have a few VMs that are a few TB in size and it takes a long time to remove their snapshots following a backup so will be great when this is resolved and the snapshots only exist for a few minutes.

Please keep us posted on any developments.

Thanks!!!!

mconte
Service Provider
Posts: 9
Liked: 3 times
Joined: Oct 18, 2013 7:51 am
Full Name: Michele
Contact:

Re: veeam 7 3par storage snapshot backups

Post by mconte »

I just noticed that the same problem affects Veeam explorer for storage snapshots too ("Create snapshots" and "rescan volume").

emachabert
Veeam Vanguard
Posts: 373
Liked: 165 times
Joined: Nov 17, 2010 11:42 am
Full Name: Eric Machabert
Location: France
Contact:

Re: veeam 7 3par storage snapshot backups

Post by emachabert »

Anton, do you have any update on this issue ?

:-)
Veeamizing your IT since 2009/ Vanguard 2015,2016,2017,2018,2019

Gostev
SVP, Product Management
Posts: 26524
Liked: 4155 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: veeam 7 3par storage snapshot backups

Post by Gostev »

No news, the case is super slow... we are escalating it, but if you can push HP from your side too, this won't hurt. The HP support case ID is 4713925829. We will be opening a couple more cases with HP on behalf of two other clients today.

phokay
Service Provider
Posts: 12
Liked: 2 times
Joined: Nov 07, 2013 9:48 am
Full Name: Michael Han
Contact:

Re: veeam 7 3par storage snapshot backups

Post by phokay »

Hi All,

Good news. I had the same problem and I looked at the issue very closely at various levels: vCenter, ESXi, Windows (Veeam Proxy), 3Par storage, etc. etc. I found that the LUN ID in question was indeed the LUN ID for the storage snapshot for the job.
The problem was that when multiple backup jobs run concurrently, in my test lab where I had only one datastores and a few VMs, Veeam tries to take multiple snapshots of the same volume concurrently as well. Guess what? the jobs sometimes failed to delete the storage snapshots (probably because of a timing issue) due to snapshots mix up on the 3Par storage array. So the workaround is to delete the storage snapshots created by Veeam on 3Par while all backup jobs are NOT running on that particular volume(s). And try to rerun the jobs. Sometimes, you will need to run full backup rather than incremental. Sometime, you will have to recreate brand new job for the VM. It works for me at the moment. When you get backups from storage snapshots working again, it's very fast and you'll be happy again.

Hopefully Veeam and HP will find the real cause and the solution very soon.

-Mike

Gostev
SVP, Product Management
Posts: 26524
Liked: 4155 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: veeam 7 3par storage snapshot backups

Post by Gostev »

The real cause of the issue discussed in this thread is well known. It is QLogic HBA driver issue which causes GetFcpTargetMapping WMI method to return an empty list. The same happens when trying to obtain a list by using fcinfo /mapping command.

Your issue is worth separate investigation, as I don't recall anything like this reported before.

phokay
Service Provider
Posts: 12
Liked: 2 times
Joined: Nov 07, 2013 9:48 am
Full Name: Michael Han
Contact:

Re: veeam 7 3par storage snapshot backups

Post by phokay »

Hi Anton,
My post was related to this error: "Backup fails with this error: Unable to find volume with node WWN XXXXXXXXXXXXXXXX and LUN ID x." as I was having the same problem. My setup was HP blades BL460c Gen8 on c7000 enclosure with FlexFabric and 3Par 7200. The FlexFabric we have is based on Emulex FCoE 10Gb adaptor. So I guess this issue is not just related to QLogic HBA.

phokay
Service Provider
Posts: 12
Liked: 2 times
Joined: Nov 07, 2013 9:48 am
Full Name: Michael Han
Contact:

Re: veeam 7 3par storage snapshot backups

Post by phokay »

Spoke too soon...it's broken again. I installed Veeam 7 on another server which has Emulex HBA Mezzanine card (not FlexFabric) and tested again. It works without any issue. So my conclusion for now is it does not work well with HP FlexFabric FCoE adaptors.

Gostev
SVP, Product Management
Posts: 26524
Liked: 4155 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: veeam 7 3par storage snapshot backups

Post by Gostev »

Since your issue was intermittent already, a single test on a different HBA cannot prove anything. It would be best to open a support case for proper investigation.

kte
Expert
Posts: 172
Liked: 7 times
Joined: Jul 02, 2013 7:48 pm
Full Name: Koen Teugels
Contact:

Re: veeam 7 3par storage snapshot backups

Post by kte »

what i also have is that i can't imprt the 3 par with his dns name , only by ip
if i use the dns name he the 3par doesn't allow is.

chjones
Expert
Posts: 108
Liked: 29 times
Joined: Oct 30, 2012 7:53 pm
Full Name: Chris Jones
Contact:

Re: veeam 7 3par storage snapshot backups

Post by chjones »

I have the exact same issue when adding the 3PAR.

We have two 3PAR 7400 4-Nodes and I added the first one fine by name, but the second I can only add via IP. I've confirmed name resolution is working just fine. Its a very strange issue. I haven't opened a support case yet but may do so just to find out what the issue is.

veremin
Product Manager
Posts: 17777
Liked: 1628 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: veeam 7 3par storage snapshot backups

Post by veremin »

Might be worth opening a ticket, indeed, as it allows us to collect as much information regarding the issue as possible. Thanks.

chjones
Expert
Posts: 108
Liked: 29 times
Joined: Oct 30, 2012 7:53 pm
Full Name: Chris Jones
Contact:

Re: veeam 7 3par storage snapshot backups

Post by chjones »

I have opened a ticked, 00476367, for this issue. I'll post back when the cause and fix is known.

chrisdearden
Expert
Posts: 1530
Liked: 225 times
Joined: Jul 21, 2010 9:47 am
Full Name: Chris Dearden
Contact:

Re: veeam 7 3par storage snapshot backups

Post by chrisdearden »

I have a customer with a similar issue - certain jobs fail when using storage snap integration. The WWN name shown when it fails is not a port name on the fabric , but the Node Name ( as against the port name).
Is it worth trying FCINFO to see if I can replicate the issue without Veeam. The Customer is using Qlogic HBA.

Case Ref: 00478986

Gostev
SVP, Product Management
Posts: 26524
Liked: 4155 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: veeam 7 3par storage snapshot backups

Post by Gostev » 1 person likes this post

Since hardware vendors are usually very slow to react to software issues, we have decided to create a hotfix for R2 that changes LUN matching logic to use SCSI Unique Id. The hotfix is currently in testing, anyone who has a support case open on this issue will get it from support once the testing is completed (if no issues are found with this approach, of course).

chjones
Expert
Posts: 108
Liked: 29 times
Joined: Oct 30, 2012 7:53 pm
Full Name: Chris Jones
Contact:

Re: veeam 7 3par storage snapshot backups

Post by chjones »

My support case about adding my second 3PAR via name has been resolved, and it still doesn't make sense why it now works.

I had created a credential in Veeam that could connect to the 3PAR (using an AD account that has full access to the 3PAR). This wasn't working. I added the 3PAR using the default built-in admin account and it worked. After it was added I switched the credential back to the Veeam specific account and it is still working.

It makes no sense why that worked, but I'll take the win.

chrisdearden
Expert
Posts: 1530
Liked: 225 times
Joined: Jul 21, 2010 9:47 am
Full Name: Chris Dearden
Contact:

Re: veeam 7 3par storage snapshot backups

Post by chrisdearden »

Gostev wrote:Since hardware vendors are usually very slow to react to software issues, we have decided to create a hotfix for R2 that changes LUN matching logic to use SCSI Unique Id. The hotfix is currently in testing, anyone who has a support case open on this issue will get it from support once the testing is completed (if no issues are found with this approach, of course).
This is Why Veeam R&D Rocks :)

emachabert
Veeam Vanguard
Posts: 373
Liked: 165 times
Joined: Nov 17, 2010 11:42 am
Full Name: Eric Machabert
Location: France
Contact:

Re: veeam 7 3par storage snapshot backups

Post by emachabert »

As soon as I get that hotfix I will post our feedback.
Thanks Anton.
Veeamizing your IT since 2009/ Vanguard 2015,2016,2017,2018,2019

emachabert
Veeam Vanguard
Posts: 373
Liked: 165 times
Joined: Nov 17, 2010 11:42 am
Full Name: Eric Machabert
Location: France
Contact:

Re: veeam 7 3par storage snapshot backups

Post by emachabert » 2 people like this post

Hotfix tested.

First results :
- storage snapshots integration is now working for backup and replication jobs :D
- when creating a storage snapshot from veeam in order to use the snapshot explorer feature it is not yet working

I will do further tests on monday to see clearly what is working and what is not.


Thanks.
Veeamizing your IT since 2009/ Vanguard 2015,2016,2017,2018,2019

emachabert
Veeam Vanguard
Posts: 373
Liked: 165 times
Joined: Nov 17, 2010 11:42 am
Full Name: Eric Machabert
Location: France
Contact:

Re: veeam 7 3par storage snapshot backups

Post by emachabert » 2 people like this post

We tested again this morning.
Everything is working as expected.

Eric.
Veeamizing your IT since 2009/ Vanguard 2015,2016,2017,2018,2019

chjones
Expert
Posts: 108
Liked: 29 times
Joined: Oct 30, 2012 7:53 pm
Full Name: Chris Jones
Contact:

Re: veeam 7 3par storage snapshot backups

Post by chjones » 2 people like this post

I got the hotfix as well from support and can confirm the 3PAR Snapshots are working for me too. FANTASTIC!!!!!

We have 6 backup jobs all with VMs on the same 3PAR Virtual Volume ... now to see how nice this plays with multiple backups and snapshots at the same time :)

Great work Veeam!!!!!

Gostev
SVP, Product Management
Posts: 26524
Liked: 4155 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: veeam 7 3par storage snapshot backups

Post by Gostev »

Thanks all for updating this topic with the results.

resruss
Service Provider
Posts: 56
Liked: 6 times
Joined: Apr 30, 2012 2:04 am
Contact:

[MERGED] SAN snapshot logic

Post by resruss »

Hi all,

We're currently testing SAN snapshots with Veeam 7 and have a question about the logic.

Let's say we've got several VMs in a folder spread across several datastores on the SAN and Veeam is targeting that VM folder
How does Veeam handle the snapshots?
Does it keep a SAN snapshot open if it knows there's another VM on the same datastore, or does it create a SAN snapshot for one VM, remove the snapshot and move onto the next VM, as it works as a VM level?
What impact does parallel processing have on VMs in a job on the same SAN datastore?
What about different SAN datastores?

Are there any risks / gotcha / best practices we should keep in mind when utilizing the SAN snapshots, especially with parallel processing in mind?

Thanks, it looks like great technology, we just want to make sure we're using it correctly and minimizin any possible risk or performance implications.

foggy
Veeam Software
Posts: 19285
Liked: 1743 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: veeam 7 3par storage snapshot backups

Post by foggy »

Please review the beginning of this topic, should answer at least some of your questions (as well as the corresponding user guide section).

Basically, a single SAN snapshot is created for all VMs stored on it. Different LUNs are snapshotted independently.

chjones
Expert
Posts: 108
Liked: 29 times
Joined: Oct 30, 2012 7:53 pm
Full Name: Chris Jones
Contact:

Re: veeam 7 3par storage snapshot backups

Post by chjones »

Well, I applied the Version 7.0 R2a update and it breaks the snapshots again with the same error "Unable to find volume with Node WWN ################# and LUN ID #". I've attempted to replace the DLLs that were provided by Veeam Support however the Proxies all show as "out of date" in the console and the console reports that the version of the console is too old. I guess we need updated DLLs.

I've logged a support case, 00499562, and hopefully there is an update available.

Post Reply

Who is online

Users browsing this forum: No registered users and 18 guests