Windows 2012 R2 Cluster + MSA 2040 FC + Off-host backup

[uros678]

Hello,

I have a Windows 2012 R2 cluster with SCV on a MSA2040 Fc SAN. Off-host proxy is the same OS version (Windows 2012 R2).

I have read in the documentation that I need a hardware VDS provider. On the HP support page for the SAN, there is no download that would have CAPI proxy, VDS and VSS included under the Windows 2012 R2 server option. And in the documentation for the MSA2040 VSS and VDS providers it reads:

"NOTE: For Windows Server 2012, VDS has been superseded by the Windows Storage Management API,
as described in “Virtual Disk Service (Windows)” at:
http://msdn.microsoft.com/en-us/library ... s.85).aspx: However the use
of VDS is still supported."

There is however a download of CAPI, VSS and VDS available under Windows Server 2008 R2.

Regarding the Veeam documentation on off-host backups, I know I need VDS hardware providers to use off-host proxy. Do I still need the package or is there some new functionality in Windows 2012 R2?

Kind regards,

Uroš

[foggy]

Uroš, I recommend you to address this question to HP support. In the downloads area for your SAN make on the HP web site I can also see the hardware VSS provider for Windows 2012 as the latest one, so I'm not sure whether it can be used for Windows 2012 R2. The most effective way would be to ask them directly what is actually required to run hardware VSS assisted backups for your storage model under Windows 2012 R2. Thanks.

[uros678]

Hey Foggy,

thanks for the answer. I was already thinking of what you suggested. Just wanted to know if anyone has any hands-on experience with this setup.

Kind regards,

Uros

[RGijsen]

I know this is an old thread but how have you fixed it? I have both P2000 and MSA2040 units with the same os and offhost proxy. I installed the 2008r2 vss / capi and it does create snapshots on the storage. But I have all sorts of issues. Do how did you proceed?

[uros678]

Hello,

I kinda just gave up on this solution and started using on-host backup. I coudn't get it to work and due to time constraint (I needed a backup) I just used on-host backup.

If you get it to work, I would be glad to know how did you do it.

Kind regards,

Uros

[jazzoberoi]

Hi Guys, HP have released new versions of CAPI, VDS and VSS drivers for Windows Server 2012 R2 - https://h20566.www2.hp.com/hpsc/swd/pub ... nvOid=4168

Can the MSA 2040 be used in direct SAN mode now ?

[foggy]

Jazz, I don't think we have this specific storage in our lab, so if you could test this and post back your findings, it would be really appreciated.

[jazzoberoi]

So, i updated the firmware on the MSA2040 and installed the CAPI, VDS & VSS drivers on the Hosts and the VBR Server.

However, i'm getting the following error :

2/21/2015 9:01:43 PM :: Unable to create snapshot HP MSA 1040/2040 VSS Provider on Hyper-V Off-Host Backup Proxy (mode: Crash consistent). Details: Provider [{9d09666c-5cf8-45ca-9294-127c95562094}] does not support shadow copies of volume [\\?\Volume{7ccd25c4-b5db-4ee1-9291-3837429c755d}\].
--tr:Failed to add volumes to the snapshot set.
--tr:Failed to perform pre-backup tasks.

What am i doing wrong ?

[Vitaliy S.]

Does the on-host proxy server work with this hardware VSS provider or you have the same error message?

[jazzoberoi]

Hi just a quick update to let you guys know that with the latest drivers / firmware from HP, the MSA2040 now works perfectly with the direct SAN mode.

[jazzoberoi]

Hi,

Now that i've been using the Off-Host direct SAN mode for a few days.. i've got a few observations..

-- The Off-Host backup (Reverse incremental) seems to be slower than doing an On-Host backup over LAN. Previously, a reverse-incremental backup would complete in 5-10 mins, but now it takes close to 20-30 mins.
-- It seems that most of the time wasted (2-3 mins) is during 'Creating snapshot HP MSA 1040/2040 VSS Provider on Hyper-V Off-Host Backup Proxy (mode: Veeam application-aware processing).' phase. Also, for some reason it only snapshots one VM at a time, even though i've checked the Proxy settings to process 4 simultaneous processes ? Am i missing some setting somewhere ?

[Vitaliy S.]

The Off-Host backup (Reverse incremental) seems to be slower than doing an On-Host backup over LAN. Previously, a reverse-incremental backup would complete in 5-10 mins, but now it takes close to 20-30 mins.

And what are the bottleneck stats for both jobs?

Also, for some reason it only snapshots one VM at a time, even though i've checked the Proxy settings to process 4 simultaneous processes ? Am i missing some setting somewhere ?

Please take a look a this topic, as it might be exactly your case > Allow processing of multiple VMs with Single-Volume

[jazzoberoi]

Hi Vitaly,

The bottleneck stats are :

Onhost Method:


Off-host (SAN Mode):


Also, I've got my Hyper-V environment setup as follows :

CSV (Lun-2) Details:
---------------------
MSA 2040 SAN Controller: A
Hyper-V Node CSV Owner: HV-2
VM Files Residing: VM-DC, VM-Deployment, VM-ERP, VM-Print, VM-Security

CSV (Lun-3) Details:
---------------------
MSA 2040 SAN Controller: B
Hyper-V Node CSV Owner: HV-1
VM Files Residing: VM-FileServer


HyperV Node 1:
--------------
VM's running : VM-DC, VM-Print, VM-FileServer

HyperV Node 2:
--------------
VM's running : VM-Deployment, VM-ERP, VM-Security


How should I organise the above to take advantage of the fastest backups processing in VEEAM ?

[Vitaliy S.]

Didn't see it mentioned in your post, but do you the same target repository in both jobs? Application aware image processing should not have such effect on your jobs. Also can just to compare apples to apples, can you create a temp job with the same VMs that would use on-host and off-host proxy servers?

[jazzoberoi]

Hi Vitaliy,

Lets ignore the speed issues for now. Can you please take a look at my cluster configuration as i've posted above and tell me if this is the optimal way to take advantage of the multi-snapshot capability in VEEAM ? Why is it making a different snapshot for each VM ?

[Vitaliy S.]

In order to have a single snapshot for VMs in the job, your VMs should reside on the same host and on the same volume. In the config above, I see that you can process VM-DC and VM-Print from the same VSS snapshot. Do you see a different behavior for these VMs?

[jazzoberoi]

Hi Vitaliy,

Yes they're all creating different snapshots. Hence i think the delay !

[Vitaliy S.]

That should not be happening given that both VMs are in the same job, host and datastore. Let me consult with our QA team on Monday regarding that.

[akselc]

Hi just a quick update to let you guys know that with the latest drivers / firmware from HP, the MSA2040 now works perfectly with the direct SAN mode.

Hi there, do you have the firmware version?
Struggling with the same issue.
Veeam B&R cannot find the MSA VSS provider...

Regards Aksel

[RGijsen]

Hi,
I'm also struggling with MSA2040 (fibrechannel though) and Veeam. 2012R2 hosts, offhost proxy 2012R2 as well. I am running the latest firmware already (GL220R005, released january 15th 2016) and CAPI and VSS driver 3.5.0.5 (januari 22th 2016), which are available for 2012R2 as well. I keep running into extremely slow 'pickups' of the snapshots. The snapshot itself is created in a few seconds, but then Veeam drops errors like:

2-2-2016 01:15:06 :: Failed to create snapshot (HP MSA 1040/2040 VSS Provider) (mode: Veeam application-aware processing with failover) Details: Unknown status of async operation
The shadow copy provider had an error. Check the System and Application event logs for more information.
--tr:Failed to create VSS snapshot.
--tr:Failed to perform pre-backup tasks.

No obvious errors show up (to me that is) on either the host or guest. Now I don't want to capture your thread but what I wonder is you are talking about VDS drivers as well. VDS is superseeded by SM-API. Still VDS drivers can be used. I can't find anything on VDS in the Veeam manuals at all. Do I need to install the VDS drivers?

[foggy]

A kind of question to contact the storage vendor with?

[itsd]

Goodmorning RGijsen,

We have the exact same issue. With the previous MSA 2040 firmware (GL210R0004) the hardware VSS Off-Host backup was working fine. Now with the new GL220R0005 we have the same errors.

Did you seek support with HP and have you received an adequate answer or a working solution for this problem?

Kinds regards,
Justus Pak
ITSD B.V.

[jazzoberoi]

Hi..

We keep getting the following too.

20/02/2016 8:01:32 PM :: Failed to create snapshot HP MSA 1040/2040 VSS Provider on Hyper-V Off-Host Backup Proxy (mode: Veeam application-aware processing with failover) Details: Failed to prepare for backup
Another shadow copy creation is already in progress. Wait a few moments and try again.
--tr:Failed to create VSS snapshot.
--tr:Failed to perform pre-backup tasks.

Can someone from Veeam please explain what is the cause for this and what can we try to fix this ?

[veremin]

Have you already checked the thread where similar issue was reported? Or raised the problem with out support team? Thanks.

[RGijsen]

Just as an update to this old thread, but still a current issue: A week or two ago I installed a MSA2052 in our storage LAN as well. For this setup I now have a MSA2040 and a MSA2052. For the MSA2052 to work I've had to update the CAPI and VSS drivers to 3.7.0.7, and now it also sees the MSA2052 volumes. However, the issue is still the same, with errors like:

25-9-2017 14:42:26 :: Failed to create snapshot HPE MSA VSS Provider on <xxxxxxxxxx> (mode: Veeam application-aware processing) Details: Unknown status of async operation
The shadow copy provider had an error. Check the System and Application event logs for more information.
--tr:Failed to create VSS snapshot.
--tr:Failed to perform pre-backup tasks.

I DO have the HyperVRestrictConcurrentSnapshotCreation key set though; so I don't run into the 'a snapshot creation was already in progress' message. Still though we keep running into these issues, while the snapshot actually DOES get created on the storage within seconds.

Since we moved to ReFS I use forever incrementals and synthetic fulls, even while I'm a bit scared not to have real full backups. But since that switch, running on-host is not an issue for us anymore as incremental backups are fairly quick for us. Still I wanted to update you that the new MSA as well as the new CAPI /VSS drivers still show the same issues in combination with Veeam.