Windows 2012 R2 Storage Spaces - Redunant proxy/repository

[Cragdoo]

As mentioned in posts elsewhere, running a Windows Server as a repository is a recommended approach. We are currently looking into implementing a redundant Storage Spaces cluster (as pictured) to provide an additional level of redundancy at the repository/proxy level. The setup we have is as follows :-





2 x Supermicro servers containing (these act as the storage 'heads') :-
• 96GB Ram
• Dual E5-2660 V2 (20 cores with hyper-threading effectively 40 cores)
• 4 x 256GB SSD RAID 10,
• 3 x SAS Cards3 x 24 disk chassis (only 8 disks in each at the moment)
• 2 x Dual Port FC HBA adapters (for Direct SAN access)
• Windows Server 2012 R2 Failover cluster and Storage Spaces

3 x 24 Disk JBOD Chassis
• Only 8 drives installed at the moment
• 6TB Disks

All 24 disks (3 x 8drives ) have been added to a Storage Spaces mirrored pool , giving 65.5TB of usable space. Currently I have allocated 32TB as a virtual disk for use.

The thinking is we can install the transport agent on both 'heads' and add both to Veeam backup infrastructure as proxies, then using some powershell to probe the cluster to determine which 'head' is the active node and has control over the storage, then amend the backup job as needed.

I will be using this post to chart the progress through the process, and welcome any thoughts/comments/suggestions.

[Cragdoo]

some direction/ideas from a quick discussion at recent Veeam TEC event in Dublin

Retrieve default location: Get-VBRJob | Get-VBRJobOptions | ForEach-Object ...
Retrieve owner of default location: http://brwilkinson.wordpress.com/2010/1 ... sources-2/
Update job to contain new location, (ie, if this location is not available, pick same repository on other server). Get-VBRBackupRepository | ? { $_.Host -Match "$repo_name" }

[Dima P.]

Hello Craig,

Here is a good MS article Storage Spaces - Designing for Performance, it includes best practices that might be useful.

[veremin]

The thinking is we can install the transport agent on both 'heads' and add both to Veeam backup infrastructure as proxies, then using some powershell to probe the cluster to determine which 'head' is the active node and has control over the storage, then amend the backup job as needed.

Update job to contain new location, (ie, if this location is not available, pick same repository on other server). Get-VBRBackupRepository | ? { $_.Host -Match "$repo_name" }

You mean you want to check which proxy server is controlling the storage at the moment and change used proxy in the job settings in accordance? Or you're willing to check storage availability, instead, and point a job to the one that is currently available?

Thanks.

[chrisdearden]

does that setup create a virtual cluster name / ip ?

[Cragdoo]

You mean you want to check which proxy server is controlling the storage at the moment and change used proxy in the job settings in accordance?

yes this , apologies I should have made this a little clearer

[Cragdoo]

does that setup create a virtual cluster name / ip ?

it does. I was thinking could we just add the cluster name/ip as the proxy, with the transport service installed on both 'heads' ? Or is that just too easy ?

Was speaking to Johan last week at the Veeam TEC and it was he who suggested the powershell method.

[veremin]

Then you can use this script that excludes all but one proxy server in the setting of a given job. The algorithm might be the following:

1) Check which proxy server is controlling storage at the moment.
2) Based on the results, enable required one, using the abovementioned code.

However, I'm wondering whether the automatic proxy selection can do the trick for you.

Thanks.

[chrisdearden]

Vladimir , what about using a virtual proxy name ?

[Cragdoo]

You mean you want to check which proxy server is controlling the storage at the moment and change used proxy in the job settings in accordance? Or you're willing to check storage availability, instead, and point a job to the one that is currently available?

Which is the preferred option ?

[Cragdoo]

Vladimir , what about using a virtual proxy name ?

So the way we have setup the cluster is

SRV1 = 192.168.1.10
SRV2 = 192.168.1.20
CLUS1=192.168.150

Can we install the transport agent on SRV1 and SRV2, add CLUS1 as the proxy/repository in B&R , and let the cluster deal with the requests for access to the proxy/repository ?

[veremin]

When you say that one of proxy servers has an active access to the repository, does it mean that the other one is shutdown or it still has connection to the storage but is sitting in the idle state at the moment? Thanks.

[Cragdoo]

Only one 'head' will be active at any one time, and in control of the storage.

[Cragdoo]

ok, my misunderstanding on the windows clustering, thinking it was smarter than it actually was. It won't combine the services/resources of both heads and present it as a single service on the cluster IP.

[veremin]

Only one 'head' will be active at any one time, and in control of the storage.

Then, automatic proxy selection should do the trick for you. The available proxy will be automatically picked up by it. Thanks.

[Cragdoo]

How will the backup handle a failover to the standby head?

[Cragdoo]

Has any one tried to create a High Availability Role in the Failover Cluster Manager ?

http://technet.microsoft.com/en-gb/libr ... MK_generic


looks like this has been suggested before http://forums.veeam.com/post119822.html#p119822

[Vitaliy S.]

How will the backup handle a failover to the standby head?

When you provision multiple backup proxy servers, Veeam backup console will choose the best proxy in terms of source data retrieval mode. If that proxy is not available (during job start) it will pick up the next one in the proxy list. This is done automatically.

Has any one tried to create a High Availability Role in the Failover Cluster Manager ?

I'm not sure it is worth clustering your proxy servers, as you can have 2-3 proxy VMs sitting in a stand by mode and waiting for tasks to be assigned in case the main proxy server goes down. As to the HA repository service, then Anton's reply seems to cover this scenario in the post you've referenced.

Hope this helps!

[Cragdoo]

So here's what I have managed to do so far.

I have added SRV1 and SRV2 as individual proxies in the B&R console. This allows me to install the transport agent onto both. I then created an HA role on the cluster, using the Generic Service option, selecting the Veeam backup proxy service then allocating an appropriate IP address(192.168.1.139) /DNS name VPXY1. I was then able to add VPXY1 to B&R as both a proxy and also add a repository REP1 (pointing to the shared volume). I started a backup job, which successfully completed. I then manually failed over the role from SRV1 to SRV2 and retried the backup job. This failed with :-

25/09/2014 09:54:29 :: Error: The RPC server is unavailable RPC function call failed. Function name: [InvokerTestConnection]. Target machine: [192.168.1.139].

I then manually failed the role back to SRV1, retried the job and it completed. I then rebooted SRV1 and immediately started the backup job again, and to my surprise the job completed. I verified that the volume and role were running on SRV2 , then waited on SRV1 coming back online, rebooted SRV2 and then restarted the backup job. Again this successful. So would appear the repository can handle a physical failure of a head but for some reason wont handle a manual role failover.

I just realised that I had been talking about high availability of the proxies, when in fact I actually meant the repositories. Apologies for the confusion.

[Vitaliy S.]

So would appear the repository can handle a physical failure of a head but for some reason wont handle a manual role failover.

I just realised that I had been talking about high availability of the proxies, when in fact I actually meant the repositories. Apologies for the confusion.

Manual and automatic "failure" should be treated in the same way, it could be some race conditions that have made your backup job fail.

[Cragdoo]

The RPC error was due to the fact I had added the cluster IP as the proxy and not the HA role IP , updated the backup job and I can now manually failover the role ( and volumes) to the other node and successfully run a backup job.

[Cragdoo]

Manual and automatic "failure" should be treated in the same way, it could be some race conditions that have made your backup job fail.

I was seeing the manual failover backup jobs fail, but not with the automatice 'failover'

[Cragdoo]

I have managed to successfully create a HA Role for the Veeam Backup Proxy Service on a 2 node Windows Failover Cluster , backed by a Shared Volume running on Windows Storage Spaces as a repository, and have tested manual/automatic failover. I have tested failing over the HA Role to the stand by node while a job was running, and the job successfully completed on the re-try. No need for SMB/CIF shares

[johan.h]

Nicely done Craig! and yes, as you're using Failover Clustering, I suppose you can just use the cluster IP address... no PowerShell needed after all

[dellock6]

I still don't get the reason to have a failover for proxies. In this way you are only using one proxy server at the time while the other is idle. With proxy roles on both servers you have double the processing capability, and still high availability since on a proxy failure, a job is assigned to survived proxy.
Anton suggested HA for the repository role (and here makes perfect sense..), not for the proxy.

[Cragdoo]

Luca , I did correct myself that the redundancy was at the repository level and not the proxy level, I was getting too giddy/excited to post coherently


I have tried a test backup using the individual proxies rather than the 'clustered' proxy, but as this isn't a Cluster Shared Volume the job then the backup failed.

I suppose I could setup a 'virutal' CSV , and mentioned in Anton's post, but this would mean using CIFs rather than the transport agent. I'm happy to trade off 1 proxy for the sake of better speed

[Cragdoo]

I'm talking rubbish ...nothing to do with CSVs in this case. The HA role only starts the proxy service on the active node. I can manually start the proxy service on the standby host and run successfully run jobs using the proxy.... more testing/investigation.

[rklingaman]

Hey Craig,

Going to bump this post and add to it. I'm working on the same project (pretty much exactly the same).

Been reading over this post a few times and little confused exactly what you did or didn't get to work.