IBM VSS-VDS hardware provider (Off-Host proxy/ Hyper-V)

[daemon]

Hi all.

I have a strange problem with IBM VSS-VDS hardware provider.

We have two sites with Veeam B&R 9.5 (running on physical servers). Each site have Hyper-V hosts in cluster (Windows 2012 R2 fail-over cluster) and Storwize v3700.
First site have FC switches and all servers are connected to V3700 via FC fabric. Second site do not have FC switches and all servers and Veeam are connected to V3700 directly.
Veeam is running backups in Off-Host proxy mode in each site. IBM VSS-VDS hardware provider is configured identically on both sites.

Code: Select all

C:\Program Files\IBM\Hardware Provider for VSS-VDS>ibmvcfg showcfg
Physical host.

cimomHost:                          192.168.1.2
cimomPort:                          5989
username (cimom):                   superuser
usingSSL:                           true
vssFreeInitiator:                   5000000000000000
vssReservedInitiator:               5000000000000001
backgroundCopy:                     0
targetSVC:                          0000001111111111
incrementalFC:                      false
cimomTimeout:                       0
rescanOnceArr:                      0
rescanOnceRem:                      0
rescanRemMin:                       0
rescanRemMax:                       45
storageProtocol:                    auto
storagePool:
allocateOption:                     standard
ioGroup:

Sometimes backup jobs on first site fails with "unknown status of async operation" error while Veeam is trying to create hardware VSS snapshot. Second site do not have such problems...
In Windows application event log on first site Hyper-V servers I can see VSS errors With ID 12298. There is one IBM KB article that describes it as "Microsoft VSS has a non configurable 10 second timeout to flush and hold VSS writers."
www-01.ibm.com/support/docview.wss?uid=swg21985096

I have IBM VSS-VDS (V3700 hardware provider) log files from both sites, and from this log files I can see that there is a difference in actions what VSS-VDS performs during VSS snapshot creation:

1) On FIRST site VSS-VDS running on Hyper-V server creates FlashCopy snapshot, after that it mount this snapshot to HIMSELF, after that it unmounts snapshot from himself, and only later Veeam Off-Host proxy trying to mount snapshot. Sometimes Veeam fails.
(host6=VSS_FREE ; host7=VSS_RESERVED ; host4=Hyper-V ; host2=Veeam ; LUN1=CSV ; LUN2=VSS_snapshot)


2) On SECOND site VSS-VDS running on Hyper-V server creates FlashCopy snapshot and after that VSS-VDS running on Veeam Off-Host proxy successfully mounts it to himself. Everything is ok.
(host2=VSS_FREE ; host3=VSS_RESERVED ; host1=Veeam ; LUN1=CSV ; LUN2=VSS_snapshot)


So I guess that this extra (and useless) mount/unmount operations are the factor what slows down everything.


Is there any one who are using Veeam in Off-Host proxy mode with Storwize (or any other array) and with hardware VSS providers?
Do VSS provider in you environment performs mount/unmount to Hyper-V server before transfer VSS snapshot to Veeam Off-Host proxy?

[Andreas Neufert]

At the site that do not work. Do the OffHost Proxy see the shared volume as offline in the disk management tool ?

[daemon]

Yes.

[Andreas Neufert]

Please create a support ticket and upload logs there. Our support team has huge experience in tracking down VSS related errors. Please add the support ticket number here so that we can look at the logs as well.

[Andreas Neufert]

Oh maybe somthing... The OffHost Proxy need Hyper-V role enabled but is not allowed to be a Hyper-V Cluster member. Is this the case?

[daemon]

Please create a support ticket and upload logs there.

Support ticket already closed. Veeam support team didn't find any issues with Veeam configuration, and they told us to contact IBM support.

Oh maybe somthing... The OffHost Proxy need Hyper-V role enabled but is not allowed to be a Hyper-V Cluster member. Is this the case?

Off-Host proxy (it is Veeam server itself) have Hyper-V Role activated. And it is not a member of Failover Cluster.
All setup is done according to Veeam documentation.

I am just curious - is it normal behavior what Hyper-V server mounts VSS snapshot to itself before transferring it to Off-Host proxy?
For example this Microsoft document says that it may happen in case of "VSS Auto-recovery".
https://technet.microsoft.com/en-us/lib ... s.10).aspx

If the shadow copy is successfully created, the Volume Shadow Copy Service returns the location information for the shadow copy to the requester. In some cases, the shadow copy can be temporarily made available as a read-write volume so that VSS and one or more applications can alter the contents of the shadow copy before the shadow copy is finished. After VSS and the applications make their alterations, the shadow copy is made read-only. This phase is called Auto-recovery, and it is used to undo any file-system or application transactions on the shadow copy volume that were not completed before the shadow copy was created.

[Andreas Neufert]

Yes, this is a correct process. Win 2008, 2008R2 and Windows 2012 do not have avhdx snapshot files. So they have to mount the snapshot to the host and revert any change between VM snap and volume snap.
Then the volume is mounted by Veeam to Offhost proxy.

Maybe you have Windows 2012 there and not R2... do you see avhdx files during processing?

Maybe there is an issue that the vss processing use somehow the WWN of the controller port connected to the Hyper-V host hard coded. So that the snapshot can not be seen by the Veeam Offhost Proxy.
I guess you have configured all connections to all hosts and Offhost proxy in a way that all devices see both controllers?

[daemon]

Yes, I can see AVHDX during backup:


All FC zones configured. Each server have 4 paths to V3700:


All Hyper-V hosts and Veeam are W2012R2:

[Andreas Neufert]

You said the Servers are directly attached to the v7000 at the non working site.
Therefore my question: "I guess you have configured all connections to all hosts and Offhost proxy in a way that all devices see both controllers?"