Hyper-V Off-Host Proxy using IBM Storewize v7000

[frowiers]

Hi

We are trying to configure our Veeam backup off-host proxy, but have encountered some challenges.

Scenario:

• Hyper-V Cluster, using CSV stored on v7000 presented via FC
• Veeam off-host Proxy, connected to v7000 via FC, with all CSVs mapped

On the v7000 we have created two Volumes which is mapped to the fictive VSS_FREE host (which essentially are all the VSS Writers installed on the Hyper-V hosts and off-host Proxy).
The names of these volumes are "VSS_Snapshot_001" and "VSS_Snapshot_002"

Demo Backup Scenario:
We have a backup job with several VMs, which are spead out on multiple CSVs.

Challenge:
When we initiate the backup, the hardware VSS starts creating flashcopy mappings, and the first backup is successful. If the second VM is on the same CSV, it uses the current flashcopy mapping, and that too is successful. When it starts the third job which is on a different CSV, it creates a new flashcopy mapping, and that too is successful.

At this stage, both of the pools are now in a flashcopy state, and therefore, the subsequent backups fail, because there are no free pools.

It seems that the VSS Hardare provider is not stopping or unmapping the flashcopy job after the backup job is "Closing Snapshot". It almost seems like the VSS Hardware Provider has initiated the wrong kind of FlashCopy Job.

This is how it looks like in the FlashCopy Window inside the v7000 console:



Does anybody know how we can tell the VSS Hardware Provider to initiate the FlashCopy with correct options, or at least make it remove the mapping after Veeam states "Closing Snapshot"?

[Vitaliy S.]

Hi Frode,

Generally, backup job should remove VSS snapshot after processing all VMS from that volume in the backup job. Queuing can only be possible if you process multiple VMs at the same time.

Can you please do the following to confirm the behavior of VSS snapshot on the off-host proxy server?

1. Create a backup job with VMs residing on the a single volume only and run the job. While the job is running check if VSS snapshot is removed at the end of the job or not?
2. Create a backup job with VMs residing on DIFFERENT volumes, but before triggering the backup job limit the concurrent tasks limit in the proxy server (for example, set it to 2 tasks at the same time). Run the job and see whether VSS snapshot is released after processing group of VMs from the first volume or not.

Let us know the results for each scenario.

Thanks!

[frowiers]

Hi, and thanks for the quick reply!

When preparing for the tests you asked me to do, I noticed that the FlashCopy jobs from the last test I did were still running (in a copy state).



I think this perhaps is the issue I am facing? That the VSS writer (either off-host proxy or hyper-v host - It's the same writer) is initiating the wrong kind of FlashCopy. I am happy to retry the tests you asked of me, but I want to wait for my current jobs to finish.

[frowiers]

Hi

I have now been able to perform the first test you wanted me to do. As you can see, the backup job succeeded:


This is how the FlashCopy status looks like now (30 minutes after the job completion):


Again, it seems like the FlashCopy mapping isn't removed after the job finishes.

[Vitaliy S.]

Frode, can you please run this command "vssadmin list shadows" on the proxy server and on the host itself? After that please post the results in this thread. This will allow us to understand whether VSS snapshots are not removed from the storage only or from Windows volume and the storage device.

[frowiers]

Hi

It looks like I have found the solution.

Got this from IBM, and that got me thinking:
IBM FlashCopy uses a unique method of copy on write with background copy.
This provides both an immediate copy of the data, and a full clone or copy after the background
copy is complete. The time it takes for the background copy to finish depends on the copy rate, and what other
loads are occurring on the storage system.

I then changed the background copy rate to 0 (which essentially means no background copy). This made the Flashcopy mappings unmap after the snapshot operation.

After I have created the Flashcopy relationship between my two v7000, I will try this in a bigger scale. Will update this post as I go on

Thanks!

[Vitaliy S.]

Yes, it might be a solution, thanks for sharing. Just wonder what is the default setting for this copy rate and when VSS/full clone is supposed to be deleted.

[phernand]

Hi, I need to make this configuration, i have Windows Failover Cluster configured with two servers and a V7000 storage connected via FC, i have a third server not a member of a cluster connected via FC to the V7000 storage to be the proxy server, in this moment the proxy server can view the volumes in the administration disk option, is offline and i hace offline shared configured, but i dont understand how to create the free and reserved volumes and how to configure the virtual host to make the proxy server can act as a Off Host proxy.

If somebody can help me whit the instructions to make this i will be grateful.

Best Regards

[Vitaliy S.]

Hi phernand,

Cannot comment on the free and reserved volumes questions, but for the off-host proxy server you should have a Hyper-V role enabled. Once this is done, VSS provider is installed and volumes are presented to that server, you should be able to run backups via off-host proxy.

Thanks!

[skrause]

To those of you using the Storwize with an off-host proxy, do you have the storage snapshot license from IBM or are you able to use the hardware VSS writer successfully without that feature licensed?

We are potentially looking at moving from VMware to Hyper-V and knowing if it will work without the licenses (which we are not going to buy) will be helpful for designing out architecture.

Thanks!

[phernand]

Thanks Vitaly, i have the HyperV Role installed, even i have virtual machines running in that server, i have configured OHBP but when i run the backup, i receive the message "there is no hardware provider" and the bakcup fails, i was read tons of articles and several of them talk about the free and reserved pools.

[Vitaliy S.]

Ok, I was under impression that you need to configure off-host proxy server in the same way as a production host, maybe there are some specifics to IBM Storewize storage.

[phernand]

Thanks Vitaly, my question is what to do this, the specific steps if you know.

On the v7000 we have created two Volumes which is mapped to the fictive VSS_FREE host (which essentially are all the VSS Writers installed on the Hyper-V hosts and off-host Proxy).

And how can i validate that is configured correctly.

Thanks.

[Vitaliy S.]

I would ask for assistance from IBM engineers, as they should know the specifics of storage configuration. BTW, does your job use this off-host proxy after re-configuration?

[phernand]

It will be great if you can have assistance from IBM, they told me that cannot help me because is not an error is a configuration issue

[Vitaliy S.]

Try to PM the original poster, as, unfortunately, I have no knowledge on the specifics of IBM storage configuration.

[tysw]

for completeness:
here's a quick guide (not mine) on how to set this up on storwize: http://www.vnotebook.ca/2014/01/configu ... proxy.html

also, i was able to duplicate the behavior that the OP describes and to fix i had to set background copy rate to zero by issuing this command to each of the vm hosts, and the proxy: "C:\Program Files\IBM\Hardware Provider for VSS-VDS\ibmvcfg.exe -set backgroundCopy 0" followed by a service restart.

[phernand]

Thank you, but the link is not working

[cloudkristen9]

There are several options to consider when discussing Hyper-V backup planning. A backup can capture
an entire virtual machine, as a point-in-time snapshot of the entire system. This type of snapshot can be
initiated from the Hyper-V manager or VMM.

However, backup software can also backup applications within the virtual machine, known as application aware
backups. Often, both methods are combined to provide the highest level of integrity and data
recovery. For most Microsoft environments and applications, VSS is the preferred and often only backup
method supported. VSS providers can be both software and hardware based. Most solutions make use of
hardware-based VSS if it is available, as it allows the storage system to do the heavy lifting, rather than
the operating system.
For Hyper-V, the IBM VSS Hardware Provider supports installation within the VM and provides VSS
snapshot capability of direct-attached volumes just like a physical server. These are application-aware
backups, using a VSS requestor and the application writer such as Exchange or SQL Server to coordinate
the snapshots. IBM provides a proven VSS backup solution with IBM Tivoli® Storage FlashCopy®
Manager.

[tysw]

Thank you, but the link is not working

ah yeah i've seen that site go up and down as i've been referring to it. I have an offline copy of it saved and i can PM you a link.
actually i just checked and the site is up again.

[phernand]

Thank you again, i have a question in this point, hope you can have some experience about.
In this moment i have just one LUN in the V7000 storage, is of about 7 TB, so is very complicate to buy disk to have two more LUNS of 7 TB for the RESERVED and FREE Lun that the document referes, the VM with the largest disk is about 580 GB, all the rest of the machines have less space asiggned, the quesion is, is absolutly neccesary to have the other two luns for RESERVED and FREE Luns of the same space of the original LUN or is enough for example with two disks of 600 GB understanding that i can send only the number of Jobs that cannot plus more than 600 GB of space?

Hope can explain clearly, and again thanks for your help.

[tysw]

you don't need separate luns for both RESERVED and FREE.
So for example:
- If you have 1 7TB CSV LUN and want to run a max of 1 concurrent veeam job against that CSV, then you of course only need 1 pre-allocated 7TB LUN for flashcopy
- If you have 4 7TB CSV LUNs and want to run a max of 2 conncurrent veeam jobs against the 4 CSVs, then you only need 2 pre-allocated 7TB LUNs for flashcopy

The pre-allocated LUNs do need to be at least thin-provisioned to the same (or larger) capacity. I have tested it a bit, and it does seem that the IBM VSS provider will try to pick the closest in size. I tried this with a mix of 100GB, 1TB and 4TB LUNs.

To get around having to pre-allocate several large chunks of disk, i would recommend moving to smaller CSV LUNs. So in your case, i would convert your CSV from a 7TB to 3x3TB CSV or 4x2TB CSV. If you have some limitation on capacity, then by splitting the CSVs up into smaller chunks also allows you to run more jobs concurrently as well.

[phernand]

Hi tsyw, first of all, thank yoy very much for your patience and the time for help me with this case.

I dont have too much experience with this kind of configuration, more specifically with the use of the VSS in the V7000 Storage, as i said in other comments, i have a LUN of 7 TB, you say the process of VSS use flashcopy, i think this means a process between luns, i have a question, if i make the backup of just one VM of 400 GB, is neccesary to have the two LUNS (Reserved & Free) of 7 TB or i can have LUNS of 400 GB?

Another question is, if i want to make a reduction of space in the current LUn of 7 TB, is possible to make this reduction without loosing data and without deleting the Lun?

Thank you

[tysw]

V7000 has no idea what kind of data is on the lun, so there is no way for it to know your single VM is only consuming 400GB of the available 7TB. When Veeam executes a job, and the hyper-v host tells v7000 to flashcopy (this is via the hardware vss provider), it flashcopies the whole lun.
and again, you don't need two "spare" luns per CSV, you just need one and it needs to be the same size as your CSV lun.

if you want to break your 7TB lun into smaller ones, you'll need to first create the smaller luns and mount them to your cluster as CSVs. Then just do live migration to the new volumes. Once you've evacuated all the VMs from the original 7TB lun, you can disconnect it from cluster, then delete it from v7000.

[phernand]

Thank you tysw, if i have in this moment just one LUN of 7 TB and i dont have more free space to make a new volume to migrate the VMs to this volume, what can i do, i think for example, to copy all the VMs to an external storage, after this, delete the 7 TB volume and create new volumes two volumes of 3.5 TB for example, one for the VMs and the other for the flashcopy, in this case is possible to create the volumes in Thin Provisioning schema?

Like you suggest is better to have more little volumes, but i have a virtual machine of 3 TB due to this two volumes is what can i do.

Another question, it is possible with the Shrink Volume option to reduce the volume space without afect the data?

Thanks in advance.

[Mike Resseler]

Can you check if you have enough space on the local disks on your Hyper-V hosts? In that case, you could do a storage live migration of the VM to the local disk (which is of course something you don't want to have for a long time...) then change the LUN and storage live migrate again to the newly lun. So the procedure would be more or less like this:

* Take a backup first but don't use the hardware VSS (this will take a bit longer but at least you know that you will have a backup in case of an issue during the procedure)
* Storage live migrate the VM to the local disk of a Hyper-V host
* Remove the CSV from the configuration, destroy LUN, create new LUN(s) to your needs, create the CSV again and add it to the cluster
* Storage live migrate the VM to the newly CSV
* Test if a backup with hardware VSS works

My 2 cents
Mike

[skrause]

Another question, it is possible with the Shrink Volume option to reduce the volume space without afect the data?

If you are talking about on the Storwize, then No.

At least according to everything IBM's docs and their people have told me.

[phernand]

Thank you very Much Mike an Skrause, i am planning how to make this configuration and your comments are very helpful!

[serverental2012]

Cannot comment on the free and reserved volumes questions, but for the off-host proxy server you should have a Hyper-V role enabled. Once this is done, VSS provider is installed and volumes are presented to that server, you should be able to run backups via off-host proxy.