Comprehensive data protection for all workloads
Post Reply
Cragdoo
Veeam Vanguard
Posts: 628
Liked: 251 times
Joined: Sep 27, 2011 12:17 pm
Full Name: Craig Dalrymple
Location: Scotland
Contact:

Windows 2012 R2 Storage Spaces - Redunant proxy/repository

Post by Cragdoo »

As mentioned in posts elsewhere, running a Windows Server as a repository is a recommended approach. We are currently looking into implementing a redundant Storage Spaces cluster (as pictured) to provide an additional level of redundancy at the repository/proxy level. The setup we have is as follows :-


Image


2 x Supermicro servers containing (these act as the storage 'heads') :-
• 96GB Ram
• Dual E5-2660 V2 (20 cores with hyper-threading effectively 40 cores)
• 4 x 256GB SSD RAID 10,
• 3 x SAS Cards3 x 24 disk chassis (only 8 disks in each at the moment)
• 2 x Dual Port FC HBA adapters (for Direct SAN access)
• Windows Server 2012 R2 Failover cluster and Storage Spaces

3 x 24 Disk JBOD Chassis
• Only 8 drives installed at the moment
• 6TB Disks

All 24 disks (3 x 8drives ) have been added to a Storage Spaces mirrored pool , giving 65.5TB of usable space. Currently I have allocated 32TB as a virtual disk for use.

The thinking is we can install the transport agent on both 'heads' and add both to Veeam backup infrastructure as proxies, then using some powershell to probe the cluster to determine which 'head' is the active node and has control over the storage, then amend the backup job as needed.

I will be using this post to chart the progress through the process, and welcome any thoughts/comments/suggestions.
Cragdoo
Veeam Vanguard
Posts: 628
Liked: 251 times
Joined: Sep 27, 2011 12:17 pm
Full Name: Craig Dalrymple
Location: Scotland
Contact:

Re: Windows 2012 R2 Storage Spaces - Redunant proxy/reposito

Post by Cragdoo »

some direction/ideas from a quick discussion at recent Veeam TEC event in Dublin

Retrieve default location: Get-VBRJob | Get-VBRJobOptions | ForEach-Object ...
Retrieve owner of default location: http://brwilkinson.wordpress.com/2010/1 ... sources-2/
Update job to contain new location, (ie, if this location is not available, pick same repository on other server). Get-VBRBackupRepository | ? { $_.Host -Match "$repo_name" }
Dima P.
Product Manager
Posts: 14396
Liked: 1568 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: Windows 2012 R2 Storage Spaces - Redunant proxy/reposito

Post by Dima P. »

Hello Craig,

Here is a good MS article Storage Spaces - Designing for Performance, it includes best practices that might be useful.
veremin
Product Manager
Posts: 20271
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Windows 2012 R2 Storage Spaces - Redunant proxy/reposito

Post by veremin »

Cragdoo wrote:The thinking is we can install the transport agent on both 'heads' and add both to Veeam backup infrastructure as proxies, then using some powershell to probe the cluster to determine which 'head' is the active node and has control over the storage, then amend the backup job as needed.
Cragdoo wrote:Update job to contain new location, (ie, if this location is not available, pick same repository on other server). Get-VBRBackupRepository | ? { $_.Host -Match "$repo_name" }
You mean you want to check which proxy server is controlling the storage at the moment and change used proxy in the job settings in accordance? Or you're willing to check storage availability, instead, and point a job to the one that is currently available?

Thanks.
chrisdearden
Veteran
Posts: 1531
Liked: 226 times
Joined: Jul 21, 2010 9:47 am
Full Name: Chris Dearden
Contact:

Re: Windows 2012 R2 Storage Spaces - Redunant proxy/reposito

Post by chrisdearden »

does that setup create a virtual cluster name / ip ?
Cragdoo
Veeam Vanguard
Posts: 628
Liked: 251 times
Joined: Sep 27, 2011 12:17 pm
Full Name: Craig Dalrymple
Location: Scotland
Contact:

Re: Windows 2012 R2 Storage Spaces - Redunant proxy/reposito

Post by Cragdoo »

You mean you want to check which proxy server is controlling the storage at the moment and change used proxy in the job settings in accordance?
yes this , apologies I should have made this a little clearer
Cragdoo
Veeam Vanguard
Posts: 628
Liked: 251 times
Joined: Sep 27, 2011 12:17 pm
Full Name: Craig Dalrymple
Location: Scotland
Contact:

Re: Windows 2012 R2 Storage Spaces - Redunant proxy/reposito

Post by Cragdoo »

chrisdearden wrote:does that setup create a virtual cluster name / ip ?
it does. I was thinking could we just add the cluster name/ip as the proxy, with the transport service installed on both 'heads' ? Or is that just too easy ?

Was speaking to Johan last week at the Veeam TEC and it was he who suggested the powershell method.
veremin
Product Manager
Posts: 20271
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Windows 2012 R2 Storage Spaces - Redunant proxy/reposito

Post by veremin »

Then you can use this script that excludes all but one proxy server in the setting of a given job. The algorithm might be the following:

1) Check which proxy server is controlling storage at the moment.
2) Based on the results, enable required one, using the abovementioned code.

However, I'm wondering whether the automatic proxy selection can do the trick for you.

Thanks.
chrisdearden
Veteran
Posts: 1531
Liked: 226 times
Joined: Jul 21, 2010 9:47 am
Full Name: Chris Dearden
Contact:

Re: Windows 2012 R2 Storage Spaces - Redunant proxy/reposito

Post by chrisdearden »

Vladimir , what about using a virtual proxy name ?
Cragdoo
Veeam Vanguard
Posts: 628
Liked: 251 times
Joined: Sep 27, 2011 12:17 pm
Full Name: Craig Dalrymple
Location: Scotland
Contact:

Re: Windows 2012 R2 Storage Spaces - Redunant proxy/reposito

Post by Cragdoo »

v.Eremin wrote:You mean you want to check which proxy server is controlling the storage at the moment and change used proxy in the job settings in accordance? Or you're willing to check storage availability, instead, and point a job to the one that is currently available?
Which is the preferred option ?
Cragdoo
Veeam Vanguard
Posts: 628
Liked: 251 times
Joined: Sep 27, 2011 12:17 pm
Full Name: Craig Dalrymple
Location: Scotland
Contact:

Re: Windows 2012 R2 Storage Spaces - Redunant proxy/reposito

Post by Cragdoo »

chrisdearden wrote:Vladimir , what about using a virtual proxy name ?
So the way we have setup the cluster is

SRV1 = 192.168.1.10
SRV2 = 192.168.1.20
CLUS1=192.168.150

Can we install the transport agent on SRV1 and SRV2, add CLUS1 as the proxy/repository in B&R , and let the cluster deal with the requests for access to the proxy/repository ?
veremin
Product Manager
Posts: 20271
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Windows 2012 R2 Storage Spaces - Redunant proxy/reposito

Post by veremin »

When you say that one of proxy servers has an active access to the repository, does it mean that the other one is shutdown or it still has connection to the storage but is sitting in the idle state at the moment? Thanks.
Cragdoo
Veeam Vanguard
Posts: 628
Liked: 251 times
Joined: Sep 27, 2011 12:17 pm
Full Name: Craig Dalrymple
Location: Scotland
Contact:

Re: Windows 2012 R2 Storage Spaces - Redunant proxy/reposito

Post by Cragdoo »

Only one 'head' will be active at any one time, and in control of the storage.
Cragdoo
Veeam Vanguard
Posts: 628
Liked: 251 times
Joined: Sep 27, 2011 12:17 pm
Full Name: Craig Dalrymple
Location: Scotland
Contact:

Re: Windows 2012 R2 Storage Spaces - Redunant proxy/reposito

Post by Cragdoo »

ok, my misunderstanding on the windows clustering, thinking it was smarter than it actually was. It won't combine the services/resources of both heads and present it as a single service on the cluster IP.
veremin
Product Manager
Posts: 20271
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Windows 2012 R2 Storage Spaces - Redunant proxy/reposito

Post by veremin »

Only one 'head' will be active at any one time, and in control of the storage.
Then, automatic proxy selection should do the trick for you. The available proxy will be automatically picked up by it. Thanks.
Cragdoo
Veeam Vanguard
Posts: 628
Liked: 251 times
Joined: Sep 27, 2011 12:17 pm
Full Name: Craig Dalrymple
Location: Scotland
Contact:

Re: Windows 2012 R2 Storage Spaces - Redunant proxy/reposito

Post by Cragdoo »

How will the backup handle a failover to the standby head?
Cragdoo
Veeam Vanguard
Posts: 628
Liked: 251 times
Joined: Sep 27, 2011 12:17 pm
Full Name: Craig Dalrymple
Location: Scotland
Contact:

Re: Windows 2012 R2 Storage Spaces - Redunant proxy/reposito

Post by Cragdoo »

Has any one tried to create a High Availability Role in the Failover Cluster Manager ?

http://technet.microsoft.com/en-gb/libr ... MK_generic


looks like this has been suggested before http://forums.veeam.com/post119822.html#p119822
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Windows 2012 R2 Storage Spaces - Redunant proxy/reposito

Post by Vitaliy S. »

Cragdoo wrote:How will the backup handle a failover to the standby head?
When you provision multiple backup proxy servers, Veeam backup console will choose the best proxy in terms of source data retrieval mode. If that proxy is not available (during job start) it will pick up the next one in the proxy list. This is done automatically.
Cragdoo wrote:Has any one tried to create a High Availability Role in the Failover Cluster Manager ?
I'm not sure it is worth clustering your proxy servers, as you can have 2-3 proxy VMs sitting in a stand by mode and waiting for tasks to be assigned in case the main proxy server goes down. As to the HA repository service, then Anton's reply seems to cover this scenario in the post you've referenced.

Hope this helps!
Cragdoo
Veeam Vanguard
Posts: 628
Liked: 251 times
Joined: Sep 27, 2011 12:17 pm
Full Name: Craig Dalrymple
Location: Scotland
Contact:

Re: Windows 2012 R2 Storage Spaces - Redunant proxy/reposito

Post by Cragdoo »

So here's what I have managed to do so far.

I have added SRV1 and SRV2 as individual proxies in the B&R console. This allows me to install the transport agent onto both. I then created an HA role on the cluster, using the Generic Service option, selecting the Veeam backup proxy service then allocating an appropriate IP address(192.168.1.139) /DNS name VPXY1. I was then able to add VPXY1 to B&R as both a proxy and also add a repository REP1 (pointing to the shared volume). I started a backup job, which successfully completed. I then manually failed over the role from SRV1 to SRV2 and retried the backup job. This failed with :-

25/09/2014 09:54:29 :: Error: The RPC server is unavailable RPC function call failed. Function name: [InvokerTestConnection]. Target machine: [192.168.1.139].

I then manually failed the role back to SRV1, retried the job and it completed. I then rebooted SRV1 and immediately started the backup job again, and to my surprise the job completed. I verified that the volume and role were running on SRV2 , then waited on SRV1 coming back online, rebooted SRV2 and then restarted the backup job. Again this successful. So would appear the repository can handle a physical failure of a head but for some reason wont handle a manual role failover.

I just realised that I had been talking about high availability of the proxies, when in fact I actually meant the repositories. Apologies for the confusion.
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Windows 2012 R2 Storage Spaces - Redunant proxy/reposito

Post by Vitaliy S. »

Cragdoo wrote:So would appear the repository can handle a physical failure of a head but for some reason wont handle a manual role failover.

I just realised that I had been talking about high availability of the proxies, when in fact I actually meant the repositories. Apologies for the confusion.
Manual and automatic "failure" should be treated in the same way, it could be some race conditions that have made your backup job fail.
Cragdoo
Veeam Vanguard
Posts: 628
Liked: 251 times
Joined: Sep 27, 2011 12:17 pm
Full Name: Craig Dalrymple
Location: Scotland
Contact:

Re: Windows 2012 R2 Storage Spaces - Redunant proxy/reposito

Post by Cragdoo »

The RPC error was due to the fact I had added the cluster IP as the proxy and not the HA role IP , updated the backup job and I can now manually failover the role ( and volumes) to the other node and successfully run a backup job.
Cragdoo
Veeam Vanguard
Posts: 628
Liked: 251 times
Joined: Sep 27, 2011 12:17 pm
Full Name: Craig Dalrymple
Location: Scotland
Contact:

Re: Windows 2012 R2 Storage Spaces - Redunant proxy/reposito

Post by Cragdoo »

Vitaliy S. wrote:Manual and automatic "failure" should be treated in the same way, it could be some race conditions that have made your backup job fail.
I was seeing the manual failover backup jobs fail, but not with the automatice 'failover'
Cragdoo
Veeam Vanguard
Posts: 628
Liked: 251 times
Joined: Sep 27, 2011 12:17 pm
Full Name: Craig Dalrymple
Location: Scotland
Contact:

Re: Windows 2012 R2 Storage Spaces - Redunant proxy/reposito

Post by Cragdoo » 2 people like this post

I have managed to successfully create a HA Role for the Veeam Backup Proxy Service on a 2 node Windows Failover Cluster , backed by a Shared Volume running on Windows Storage Spaces as a repository, and have tested manual/automatic failover. I have tested failing over the HA Role to the stand by node while a job was running, and the job successfully completed on the re-try. No need for SMB/CIF shares
johan.h
Veeam Software
Posts: 711
Liked: 178 times
Joined: Jun 05, 2013 9:45 am
Full Name: Johan Huttenga
Contact:

Re: Windows 2012 R2 Storage Spaces - Redunant proxy/reposito

Post by johan.h »

Nicely done Craig! and yes, as you're using Failover Clustering, I suppose you can just use the cluster IP address... no PowerShell needed after all :)
dellock6
VeeaMVP
Posts: 6137
Liked: 1928 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Windows 2012 R2 Storage Spaces - Redunant proxy/reposito

Post by dellock6 » 1 person likes this post

I still don't get the reason to have a failover for proxies. In this way you are only using one proxy server at the time while the other is idle. With proxy roles on both servers you have double the processing capability, and still high availability since on a proxy failure, a job is assigned to survived proxy.
Anton suggested HA for the repository role (and here makes perfect sense..), not for the proxy.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
Cragdoo
Veeam Vanguard
Posts: 628
Liked: 251 times
Joined: Sep 27, 2011 12:17 pm
Full Name: Craig Dalrymple
Location: Scotland
Contact:

Re: Windows 2012 R2 Storage Spaces - Redunant proxy/reposito

Post by Cragdoo »

Luca , I did correct myself that the redundancy was at the repository level and not the proxy level, I was getting too giddy/excited to post coherently ;)


I have tried a test backup using the individual proxies rather than the 'clustered' proxy, but as this isn't a Cluster Shared Volume the job then the backup failed.

I suppose I could setup a 'virutal' CSV , and mentioned in Anton's post, but this would mean using CIFs rather than the transport agent. I'm happy to trade off 1 proxy for the sake of better speed
Cragdoo
Veeam Vanguard
Posts: 628
Liked: 251 times
Joined: Sep 27, 2011 12:17 pm
Full Name: Craig Dalrymple
Location: Scotland
Contact:

Re: Windows 2012 R2 Storage Spaces - Redunant proxy/reposito

Post by Cragdoo »

I'm talking rubbish ...nothing to do with CSVs in this case. The HA role only starts the proxy service on the active node. I can manually start the proxy service on the standby host and run successfully run jobs using the proxy.... more testing/investigation.
rklingaman
Influencer
Posts: 21
Liked: never
Joined: Feb 22, 2010 6:18 pm
Full Name: Ryan Klingaman
Contact:

Re: Windows 2012 R2 Storage Spaces - Redunant proxy/reposito

Post by rklingaman »

Hey Craig,

Going to bump this post and add to it. I'm working on the same project (pretty much exactly the same).

Been reading over this post a few times and little confused exactly what you did or didn't get to work.
Post Reply

Who is online

Users browsing this forum: Google [Bot] and 157 guests